I think there are only two long term solutions:
- Introducing some kind of proof of work -- e.g. you do work to downvote/upvote;
- Some very "localized" reputation -- e.g. you trust the friends of your friends more.
Those ideas are behind the Bitcoin protocol and the Web of Trust, respectively.
In this case each would have it's problems: proof of work is inefficient by design and needs a good hash function not to be exploited; local reputation by design makes it hard to find new/unrelated content.
Having no reputation wouldn't mean you can't do anything -- just that what you do is less trusted and has lower priority.
So if you want to take the Pirate Bay approach of providing a small authoritative list of "trusted moderators" through the web and have all the users use that filter, you have to keep it simple so that it can be easily replicated if the web-component gets taken down.
Complicating the web/authority component means mirroring it and resurrecting it becomes hard.
I find this stuff very interesting, let me know if I'm missing an obvious approach.
The hard part is the new-user bootstrap. They need a starting list of trusted people - as they pull down information from the swarm, they get a good picture of the "web of trust". Then they can look for signed upvotes or downvotes on any content they're researching - that is, instead of just looking for the latest Game of Thrones vid, the system also looks for all upvotes and downvotes about each particular copy of the latest Game of Thrones vid and compares it against the user's personal "web of trust" to know which votes are respectable. The user sees this as just "here's the most popular copy".
Of course, passing around millions of upvotes and downvotes P2P would not be an easy problem.