RIP Google PageRank score: A retrospective on how it ruined the web

Ultimately Google's ability to quantify both quality and relevance will converge on actual quality and relevance. In other words, the way to "game" the system and practice SEO will be to create a site that is actually relevant with high-quality content. But we're not there yet, and who knows how close we'll actually get.

It's not nearly enough to offer a better product at a better price. Our company, which was objectively superior to the bad imitators that followed us, was eviscerated by a competitor with a pedigree as a professional spammer -- his product rarely worked and we were constantly getting refugees once they were able to wade through the massive amounts of spam that promoted his solution and find some genuine information. It didn't help that he bought off niche webmasters to delete any mention of our service from their forums and only allow mention of his service.

I have accepted it as a reality that "SEO consultants" (read: professional spammers) are a necessity if your site is going to get anywhere. Note we've always been SEO optimized, meaning our site was always search engine friendly and it contained a great deal more relevant content than our competitor's site. But because our competitor enlisted his network of spam websites to manufacture backlinks and actively sought to exclude mention of my product from the internet by paying off webmasters, he performed much better.

People don't know and generally don't care if your product is better or worse. If you have a product that looks like it's functioning, that's all the development that needs to be done as long as it half-works 5-10% of the time. The rest of the money needs to go into spa--err, sorry, "internet marketing".

What has been Google's saving grace has been their skill at updating their target, attempting to turn the chaos and effort from SEO towards improvement in content. They have made an impossible problem into a mostly-solvable control systems[2] problem.

[1] https://en.wikipedia.org/wiki/Goodhart%27s_law [2] https://en.wikipedia.org/wiki/Control_system

Brad Templeton made the observation way early on that his humor mailing list went to over ten million people. If people sent him a penny if they laughed, and only 1% did, he would make $30,000 a month or $360,000 a year just for curating jokes. Scale + pennies can add up to big dollars on an individual level.

PageRank, AdSense, AdWords, all fed this mechanism with a way to turn a small amount of work by an individual into disproportionate returns. It really has been no surprise that the web spam problem became much much worse than the email spam problem. There was a better built in payment mechanism.

True, but they very often don't beat out numerous websites that have taken the open StackExchange content (or other similar, more specialized forums) and rehosted it on a new domain. This behavior should be easy to kill, Google should consider this blacklistable behavior and have an easy way to report any site doing it, with the number of StackExchange users out there these sites would be blacklisted within hours of being launched and before long peolple would simply stop doing it. But for some reason it is allowed to continue.

Do you remember when Yahoo was a website about wrestling ?

Or when having a good section in DMOZ was important ?

With Google, I feel like I have to fight with it. If I'm searching for something obscure or perhaps a word that is misspelled on purpose it thinks it knows better what I'm looking for. It also often returns searches without the word that I searched for and often ignores when I prefix it with + or put in quotes.

I'd like to see a hybrid of the high-speed algorithmic scanning of the pages that we see now combined with an army of human reviewers, including super-reviewers who are recognized industry experts, who periodically rate indexed content based on quality. Backlinks and other derived consensus measurements should be given far less weight and a combined algorithmic and human quality rating should be of at least equal importance.

So I want to go back to the days of a manually curated web index, combined with the technology needed to make that span out over billions of web pages.

PageRank was one part of the reason we have a search as powerful as Google, the use of links to assign an authority score. Another big part was the use of link context -- which isn't part of PageRank but part of the overall search algorithm at Google.

PageRank score is when Google revealed those scores to the public for any page. That's not something it had to do, in order to use PageRank as part of it algorithm. But in doing so, it fueled an explosion in link spam.

But I will actually claim that PageRank itself is a problem not just because it is so gamed but also because web page authors link to things they find via Google, creating a positive feedback loop that undermines the very premise of the PageRank algorithm.

Search engine research had stagnated because of Google's dominance, and Google itself is not motivated to change, much as Microsoft was unwilling to evolve its cash cow. Rather than innovate in search, it spends most of its resources on ways to shore up its dominance (Google+, Android), fighting the very thing its search engine feeds (Internet garbage), and strengthening its advertising business.

If you "never used the web before Google," how would you know this? I suppose you might have read about it. I did use the web before Google, and I don't remember web rings back then. There simply was no reason to do so, before Google started ranking pages based on the links.

I do remember lots of BS meta keywords.

The article isn't about Google's influence on the web, as a whole. Google has had a huge influence on the web in many ways, from making it easier for people to locate information to sites considering how to speed up their content, to become mobile-friendly or to use secure connections, because Google rewards such things with ranking boosts.

The article was specifically about PageRank's influence on the web, in terms of link brokering and link spam. Before Google released PageRank scores, some of this happened. It would have happened even if scores had never been released, because it was well-known that Google leveraged links and thus, links had value.

But PageRank scores were an accelerant. They allowed people to use Google's own scores to assign value to pages, value that could be translated into monetary value. It really did reshape the link economy, to the degree that we had a court case with a First Amendment ruling on Google's search results (amazing, when you think about it) as well as an entire new standard to restrict the credit links could pass, nofollow.

I doubt Google anticipated this. Showing the scores, as the article explains, was meant as an incentive for Google Toolbar users -- "Hey, enable this feature, and we'll show you how valuable a page is deemed to be." Google's gain, of course, was that anyone enabling this sent their browsing patterns back to Google, so it better understood what was happening on the web outside of its own properties.

The unintended consequence was that PageRanks scores fueled an explosion in link buying and selling, as well as link spam.

Also, I didn't say the web was better before Google. It had plenty of problems, though it wasn't "pretty much un-navigable pre-Google," as you say. Many people used many of the search engines that were bigger than Google successfully for years. If it were really that bad, by the time Google came along, people would have given up on the web.

Google, of course, was a huge improvement in search and for the web as a whole. The article wasn't that Google was bad for the web. It really was just focusing on one aspect that didn't help the web, how releasing PageRank scores ironically fueled some of the spam Google has to fight (and which it fights well) as well as the spam third-parties have to deal with.

https://en.wikipedia.org/wiki/Goodhart's_law

A friend mention that recently to me and it was really eye-opening. Once you start looking, the pattern is everywhere.

This.

Google are not interesting in protecting us web users from SEO, but themselves (or more specifically their core ad business). After all why would you pay for Google ads if you could just get free traffic from Google via SEO techniques? So SEO is the logical competitor to Google.

When Google arrived, it was a huge advance in search. It offered an obvious improvement in relevancy, which is why so many serious searchers switched to it from AltaVista and then users of other search engines moved over.

Nothing I wrote suggested that Google was bad, didn't offer great relevancy or anything like that.

My story is about what happened when Google revealed PageRank scores for pages across the web. That fueled an explosion in link buying and selling. It allowed people to attach Google's own score to a page, a value if you will that Google itself placed on those pages, which made it easier to then assign a monetary value.

In turn, that lead to many of the woes that the web as a whole has to deal with today: understanding how to use nofollow to block links, to stay in Google's good graces. Spam mail pitching links, trying to buy links. Link spam

I'm sure we'd have had some of this even without PageRank scores ever having been revealed. Perhaps it would have been as much, even. After all, it was well-known that Google was leveraging links as part of its ranking algorithm. The market would have been there.

But I do think that releasing the PageRank scores accelerated market faster than it would have done otherwise.

Back to the gaming -- again, it feels like you're reading stuff I didn't actually write. I'm certainly not saying that Google itself introduced the ability for people to try and game search engines. That was happening even before Google existed. Of course, Google initially thought it was immune. In 1998, Sergey Brin even said this on a panel that I moderated:

"Google’s slightly different in that we never ban anybody, and we don’t really believe in spam in the sense that there’s no mechanism for removing people from our index. The fundamental concept we use is, you know, is this page relevant to the search? And, you know, some pages which, you know, they may almost never appear on the search results page because they’re just not that relevant."

Google soon changed its view and introduced extensive spam fighting efforts. Those were inevitable. As you say, it was prey that would attract predators. And even with the link selling, it has done an admirable job fighting off the spam. It's not always perfect, but it's a very robust system.

Nevertheless, the spam attempts will continue regardless if Google actually blunts them because, as the article explained, there's simply so many people with misconceptions that they'll chase anything anyway. PageRank scores fed into this, that's all.

The value of "ch" is a checksum you have to precalculate.