Perhaps they could do as on Stack Overflow, and let a downvote cost one point from your accrued points? Or perhaps that's too much of a re-design. It could regulate spam downvotes though. (I think the biggest problem with SO is that your point pool is visible, with gold and silver and so on, which leads to anyone with a high enough sum being treated differently, although that might actually not reflect competence or expertise.)