"As a result" is a pretty big stretch. Traffic accidents result from unexpected situations. People who drive a lot learn to read the environment and the drivers around them and anticipate what will happen next.
When those anticipations are violated, it runs the risk of creating a surprise that can't be managed. That is just as true for a violation in the conservative direction as in the aggressive direction.
Learning to drive is an empirical process based on observed behavior, not a deductive process from the rules of the road. So I'm not surprised that self-driving cars are disliked. They don't drive like people, and thus violate the heuristics that human drivers have built up to predict traffic behavior.
And perhaps more importantly, they look different, so they can feed into confirmation bias about how annoying they are. If it was impossible to distinguish self-driving cars (if they exhibited the same visual diversity as cars in general), it would be harder for people to build mental associations about them.