You would have 4 options for each item, ( Upvoted, Downvoted, Not Voted, and Blocked). If a SEO manipulator is Upvoting a bunch of spam content that you didn't vote(that you've seen) for, down-voted, or blocked, they would quickly un-pair with you. The next step for SEO is for them to start voting identically and predicting to how you would vote so that a few paid ads didn't skew the results to much, but isn't that mission accomplished[0]?
This isn't done because it's grouping the users and not the content? It doesn't scale? And yet thematically this seems close to the model used by Google FLoC. If Google would just make a sub-reddit like system for each cohort so that we could submit and discuss content, I have a feeling that would be very successful and people would opt-in.