In addition to accident-type safety issues, I also find driving a bit worrying from the perspective of what you might call "legal safety". The chances of ending up in a negative interaction with the police seem much higher when you drive than when you travel via other modalities, because a traffic stop has a pretty low bar of evidence, and is often used as a lead-in to various other searches/etc. I probably am realistically not at that much risk (being a white middle-class person usually traveling in middle-class areas), but I still feel less exposed to the police when not driving.