Well, even so, while most drivers don't mind driving, there is a significant minority who are actually kind of freaked out by driving -- big overlap with people who have experienced a serious wreck. I've met a few people like that.
And besides the ones who are so anxious about driving that the rest of us know about it, there's probably a lot of people who have mild anxiety about driving but just suppress it, get through it, and/or minimize unnecessary driving, because you can't avoid driving where they live.
And, considering the risks, it's not an irrational fear. Perhaps it's us who don't mind driving and are acclimated to it who are the strange ones.
Heck, I do find myself feeling this way sometimes. I've been driving long enough to be pretty comfortable, but I do notice feeling a lot more comfortable on safer feeling roads, eg with slower speeds, frequented by less hurried drivers. I think everyone does, to some degree, otherwise complaining about the crazy drivers from the next state over wouldn't be such a universal pastime in America :)
We pretend the risk and the associated rational anxiety of driving doesn't exist because driving is so commonplace, but it's there and it's pretty big.