Voting would likely require being logged in which is not so good for the more privacy orientated search engines.
Could flip the idea on its head and look at ways to verify the owner of a site/page (again, with privacy issues associated), and use that as a quality metric.
To what end? A verified real identity could post crap. Meanwhile an anonymous actor could publish gems. It comes down to how much trust you place in a given publisher; that currently goes by domain. The only thing verified identity buys you is the ability to trust snippets from that person published across a variety of untrusted domains.
Meanwhile, analyzing per-account feedback patterns along with other session behavior significantly raises the bar for abuse.
The general idea would be that people are harder to fake as a signal than links.
A lot of sites associate social profiles with their domain/page. Something like a topical author Pagerank would be interesting, with endorsements. That too, of course could be bought and gamed. Pagerank was a game changer when it arrived, essentially acting as a 'vote' for a page, where some votes are more important than others. Associating votes with people (like the OP thought) is interesting.
I don't think it would be as hard as you think. There are already "captcha farms" in India and Bangladesh - "Identity farms" would be just as simple to set up.
You would have 4 options for each item, ( Upvoted, Downvoted, Not Voted, and Blocked). If a SEO manipulator is Upvoting a bunch of spam content that you didn't vote(that you've seen) for, down-voted, or blocked, they would quickly un-pair with you. The next step for SEO is for them to start voting identically and predicting to how you would vote so that a few paid ads didn't skew the results to much, but isn't that mission accomplished[0]?
This isn't done because it's grouping the users and not the content? It doesn't scale? And yet thematically this seems close to the model used by Google FLoC. If Google would just make a sub-reddit like system for each cohort so that we could submit and discuss content, I have a feeling that would be very successful and people would opt-in.