Well, obviously PGP has the whole "web of trust" thing but that's a bit hard to use. You can use crypto to "sign" things, so simply "signing" your upvotes and downvotes attaches a person to a vote. Then you let users upvote and downvote
other users building a web of trust and a black-list. Signed-upvotes and downvotes would get passed around in the swarm itself as content.
The hard part is the new-user bootstrap. They need a starting list of trusted people - as they pull down information from the swarm, they get a good picture of the "web of trust". Then they can look for signed upvotes or downvotes on any content they're researching - that is, instead of just looking for the latest Game of Thrones vid, the system also looks for all upvotes and downvotes about each particular copy of the latest Game of Thrones vid and compares it against the user's personal "web of trust" to know which votes are respectable. The user sees this as just "here's the most popular copy".
Of course, passing around millions of upvotes and downvotes P2P would not be an easy problem.