Of course part of the answer is that new comments have no votes so a quick downvote will make them grey, but there are frequent strange cases.
One thing for sure is that a lot of people downvote based on disagreement rather than a comment's quality - which in my opinion is not right (and I think against the intent of a downvote), but that's a different issue.