A lot has changed since then, but I doubt that Google has decided these kinds of content farms are suddenly OK. It should also give pause to people asking for "open ranking algorithms." Without some amount of secrecy, content farms will pop up for literally every keyword they can sell ads against, and they'll know how to rank more often rather than just blindly guessing.
I have a feeling there is a way to introduce randomness to this search engine in a way that smooths out the game theoretic arms race of gaming an algorithm. This goes against our natural urge to have a fully deterministic algorithm which always brings the most pertinent sites right at the top. However, because humans are trying to bring order to things for their own benefit, throwing in chaos could prevent people from even trying to game the algorithm
Introducing random elements just makes it worse for everyone.
The task of search has gotten exponentially harder as well. There are billions more website pages, more spam, more sophisticated attacks, more gaming of the system, etc. They could have gone from a 80% relevant result to a 99% relevant result and you perceive at 10% drop in relevancy. It's a hard problem.
Google is the goto search engine today, the competition is virtually non-existant, so they don't have to compete anymore. Hence the quality of their search engine going down hill, conflict of interests and what not. It's not shortsighted, it would take billions to compete with Google search and even Bing isn't even trying anymore.
When I look at a Google search page result today I see loads of ads, then their shopping stuff, image search stuff, a list of questions related or not to my search and their answers extracted from third party web pages, then only when I scroll 1 page I see actual search results. Google search wasn't like that 15 years ago.
Googlers are working so that Google makes more money, and Google is an ad company first and foremost.