NEVER downvote:
1) Just because you disagree with a post.
2) Because you dislike the poster.
3) Because others have downvoted a comment into oblivion and you want to jump on the bandwagon. i.e. No need to kick a comment when it's already down.
4) When you have not explicitly reasoned out why you are downvoting beforehand.
DO downvote:
1) Because the comment is derogatory or offensive.
2) Because the comment is factually incorrect.
3) Because the comment is off topic and/or does not add to the conversation.
4) Because the comment lacked any meaningful content. Ex: "Yes!", "Nice post!", "LOL!", etc.
Ultimately the golden rule applies here. Do not downvote any comment for a reason that you yourself would be upset over if someone used that same reason to downvote a comment of yours. In short, just put some thought into it.