I like the reason "This doesn't add to the conversation."
My rule of thumb is to upvote if the comment is insightful or has information I find useful. Downvote if it seems blatantly incorrect, mean, overly negative or is actively trying to make the conversation worse. Otherwise I just ignore it.