The more direct response to the OP would be to limit downvoting based on whether a user has already downvoted another post of the same user already. But even that gets silly.