Why not just create a user curated search engine?

I've been reading a lot of the Google-related search engine complaints. Who hasn't? They're everywhere for whatever reason. A common theme that I seem to see in most of them is that people seem to agree that some sort of user interaction is one of the few ways to eliminate spam and SEO content farm results.

So, why not just create a user curated search engine? Web pages would be indexed. Users would filter them, using queries. On a per-query basis, indexed pages would be subject to a reddit-esque voting system for relevance to that query. There would be no visible up voting, though. There would only be a visible down vote option. Up voting would be done automatically when a user clicks a result. Down voting would be done if a user visits a page and the page is not relevant to the query, so they come back to the search results and down vote the result (it would remove the automated up vote + 1 more to net a subtraction). Pages indexed more recently would be given priority over older pages. So what would eventually happen is the most viewed and newest pages for a query would bubble to the top, and older posts would be in the middle, and down rated posts would be at the end or maybe even excluded all together.

Obviously there are holes in this, such as people scamming the up-vote clicking mechanisms but I'm sure that could be prevented with more thought.

I think the key is that all of this voting is done on a per query basis, for relevance to that query.