Hacker News Reader

Comment by nmstoker | Hacker News Reader

I don't know why Google doesn't grasp the nettle and just not index pages like those super low quality ones (ones that no human is ever ever going to want to view). They could surely infer them via a series of experiments: temporarily backing off on ads to a particular user and see which sites they click through to.

They needn't hit their ad revenue much at all, because each user would just be in an experiment briefly, and they would aggregate results to get a general view.

View on HN

From Google's point of view, it's working exactly as intended. Once they've captured market share, there's no reason not to turn the screws and squeeze out every last drop of revenue. Google would love for the internet to consist entirely of ads. Other content is only necessary insofar as necessary to lure in people to see the ads. And in Google's case, just the hope they might find content is enough to suck a user into the ad vortex.

Perhaps this isn't a great long-term strategy, but that doesn't really matter because nobody involved is incentivized to think beyond the short-term metrics that determine their bonus/promotions.

What's truly amazing about this whole situation is that Brin and Page literally called it:

http://infolab.stanford.edu/pub/papers/google.pdf

Everyone—Everyone, who mattered (plus many others who didn't) knew exactly what would happen as Google went deeper and deeper into ads. But it made line go up real well, so....

They used to. For most of the '00s Google'd noticeably exhibit a kind of wave pattern: webspam would creep onto page 1 of results, more and more, then abruptly vanish. Repeat. Cycle of, IDK, a few months.

To all external appearances, Google simply gave up on this cat-and-mouse game around '08 or '09. Big sites get ranked high (whatever changed seemed to give those a huge boost, around that time), everything past that's mostly webspam.

What's the upside for Google? They get paid regardless with the injected top results as ads, as long as there's no competitor threatening their market share, the quality doesn't matter anymore.

They prefer to spend money to keep the monopoly going such as the Safari deal.

96% of the web already is not indexed. Clearly there is something wrong with the algorithm they use to determine what should be indexed and what shouldn't. All that AI doesn't seem to help them much

If i may ask where did you get this number? How are the other search engines on it compared to google?

I have no idea where they're getting that number from. The relevant search terms is "Deep Web" (not to be confused with the Dark Web).

https://www.sciencedirect.com/science/article/abs/pii/S03064...

The abstract from that article has a good introductory summary.

For me, the problem is what makes a webpage super low quality. The results I'm tired of seeing all contain the same watered-down language as the other top results that they either copy outright or run through a thesaurus.

Because those crap pages are filled to the brim with Google Ads that make up the most of their revenue, it's a conflict of interest.

Reply on news.ycombinator.com