The logic is that you could've saved the person but neglected them instead leaving them to die.
In the US, you are protected under the law if you attempt to help someone in need but injure them, but many people don't care enough to help, don't know how to help, or are afraid to be sued in this sue-happy country. (Even if you tried to save a life and failed, you're fucked if you're taken to court and found guilty of murder and can't sufficiently prove you were being a good Samaritan.)
Off-duty doctors also won't usually get involved because good Samaritan laws don't always sufficiently protect them against potential malpractice accusations. If a doctor loses a malpractice case, they'll face fines, jailtime, or even losing their medical license. This isn't an issue for good doctors working within a hospital and having their medical equipment, but on the street on a day off without all their medical supplies? They're fucked in a region that makes no distinction between helping someone on the clock vs off the clock.