You need a certain (small) amount of karma before you can downvote people. At that point, you'll see a downarrow below each comment uparrow.

That was done to prevent people from creating new accounts for the purpose of repeatedly downvoting someone.

You need to reach a certain "Karma" threshold. (I believe it is something in the range of 50-100)