Actually it's because so many cyclists break the rules that do apply to them: running red lights, cycling on the pavement (sidewalk), and so on. And this is why pedestrians often feel the same animosity towards cyclists that drivers do.
Drivers aren't hating cyclists because one group breaks the rules, as if the other doesn't.
The actually problem isn't that cyclists break laws. It's that they break them doing incredibly stupid and dangerous things that I don't/can't expect. I know most of the stupid things other drivers are going to do. Cars are easily visible and I can usually see stupid maneuvers coming. I've had several close calls with cyclists. A common scary scenario is when they are driving against traffic on the sidewalk when there are serious visual obstructions. That's what makes me the angriest.
At speed. Don't forget how fast they buzz along now that they don't have cars to worry about, and how quietly they move.
I greatly prefer skateboarders. Even if they move at a good clip, they're clattering along and giving everyone plenty of advance warning. Bikes just barely make any sound at all until they're nearly on top of you.