You need more karma. There's a karma threshold below which you can't downvote. I haven't reached it either. Soon, though. And when I do. Oh...