You need to have enough HM karma to see the downvote button. A long time back it used to be 100, no idea if it still is. Have an upvote to get you a step closer.
You'll get a downvote at 500 karma.
I never downvote. Used to, but stopped.
I’m quite liberal with them, I think it’s one of the things that stops this place turning into Reddit.