Why does it bother you? Probably only because you've been taught that you're supposed to hate it. There's nothing inherently wrong about vocal fry: there are human languages where vocal fry is an essential component of native speech.
I don't think it has anything to do with some vague notion of "hate".
Some people simply find it to be a distracting phenomenon that can make it harder to understand what somebody else is saying, even when they like the speaker.