One thing you miss here is the asymmetry of risk. Cyclists are at highest level of risk while on the road. They're moving at enough speed to cause serious injury to themselves, but they're still usually about half the speed of cars. Modern automobiles have excellent safety characteristics and pedestrians have more of an ability to take evasive action.

A mistake on a bike is going to almost always be a life-or-death situation as just falling onto the road puts them at extremely high risk for injury. The cost of errors made by car drivers has a huge impact on the morbidity of cyclists, while the opposite doesn't hold true. Even if cyclists were an order of magnitude more dangerous than car drivers, their poor decisions would have negligible impact on car drivers.