Downvoting is also restricted to accounts who exceed a point threshold. I think that particular feature (ensuring people who can downvote at least have some level of trust within the site) has been critical to prevent hive mind-style downvoting. I rarely see downvoted comments where I don't understand why they were downvoted.