IMO there shouldn't be a down vote button at all, and to be honest I can't see its purpose. If a comment is trolling or being aggressive then you should flag it, so where is there room for the down vote button?
It becomes quite intuitive to just use it for what you consider an "invalid opinion" regardless of how much you try to teach people otherwise, because it's so convenient to use and people are less rational when they are angered.