This seems to nail it based on my experience.

At least anecdotally, the people I know that are vaccine-averse have the same fear of airplanes, etc.

It might just be the way they reason, fixating on worst-case outcome instead of a more balanced view.