You'd probably have to have a trusted human-in-the-loop. I.e., if a site gets enough down votes, it would trip some internal procedure to review the site.

I think also if you didn't generate public feedback (i.e., you didn't list the most down voted sites for example) you'd avoid the kind of Beiber-to-DPRK stunt that 4chan would be interested in.