But a jerk walking is virtually certainly not going to kill me (or my kid). A jerk cycling almost certainly won't either. But a jerk driving quite possibly will - after all, where I come from there's a 9/11 on the roads every month due to drivers, and the dead are disproportionately people walking or cycling.
Of course, 100 years ago the streets were for walking (or horses, bikes, early autos, whathaveyou) and it was quite a feat that auto makers convinced people that only folks inside cars deserved to use 90% of it (all of it, in much of the US), and that it was perfectly normal to leave people walking clinging on to a shred of space at the edge.
And keep in mind, it's a lot more dangerous for a woman to walk alone than the average HN poster, especially if they need to do so at night.
“Mugging” can't happen in a car only in the sense that when it happens to people in a car, it is called “carjacking” instead of “mugging”.
When I cycle I fear being killed by a motorist; when I drive I fear killing a cyclist. It's important to clarify whether cyclists are putting you in danger or if they are stressing you out. Both suck, but one sucks a lot more.
You are quite right in terms of where the danger lies though. It's on the driver more than any other to be the most careful with their actions.
You can see the same behaviour from car drivers when the rules don't make any sense (as for example in some upcoming countries).