Ask HN: Does minimum karma downvoting encourage elitism?

When I started on HN a while back I had just assumed the visible lack of downvoting arrows meant that feature simply didn't exist. I preferred it, since to me it meant that quality comments would bubble to the top, while poor arguments would be ignored.

Then I hit the 500 karma mark and realized it was a privileged action. From time to time I'll post a comment that gets downvoted without explanation, and I have to wonder what is being accomplished by signaling disagreement without standing up for anything.

I would like to hear your opinions, as I believe what happens is that when people are designated as part of a privileged class, they will think themselves to be smarter representatives of the community, such that it acts as a justification to downvote and stifle discussion without having added anything to it.

I'm not interested in gaining imaginary internet points, so I have to wonder what the purpose for downvoting is in the first place. If the comment is inflammatory, off topic, abusive, and so on, we have flagging mechanisms for that. So in that case, why bother downvoting?

(Yes, I realize a major reason to have minimum karma requirements is to prevent trolls and bots)