Are you confusing this with topic downvotes? They don't exist. Do you know where to look for the comment downvote link? It's a small gray arrow under the upvote arrow just to the left of the name of the commenter.
If none of this applies then I can't help you.
I wish I could turn off the downvote arrow. I never use it, and I have to be careful not to click it by mistake when upvoting.
I usually find that even here on HN, the symptoms people report are not actually what's happening, but are filtered through their incorrect assumptions. It is important to report what you see, not what you think is happening underneath. It's so easy to fall into the trap of reporting what you think must be happening.
Congratulations on completing karma level five hundred. From now on, it's just karma for karma's sake.