When it happens to me it is because I have replied to that comment. I assume that it is because if you replied to a comment you DO think it is worth reading it, and it surely contributed to the discussion, therefore no sense in downmodding it
I've seen it occasionally go awry - you'll upvote and the vote just disappears.
I find I can vote on comments I reply to... (it doesnt, to me, appear to be logical that if your replying you agree with the post).
I can still down mod the comments in this thread that I have replied to.
EDIT: wait NOW I see :) I cant down the above comment. Ignore me :)
I volunteer to be downmodded and replied too :)