Hacker News Reader

My impression back when Google really took off was that other search engines were basing their relevance scores on page content and getting lots of spam in their results from pages that gamed the relevance algorithm. Then here comes Google using human judgment via PageRank and getting great results, because who would link to garbage?

Of course the web has changed since then. And I think you're right on in thinking that the signal-to-noise ratio of links has gone way down. But it may not be fair to chastise Google for not getting good results just from page content. Has anyone ever done a good job of information retrieval in an adversarial environment without leaning on human judgment?

I agree with much of this.

I believe the antidote to this is simply acknowledging small communities curate the best content, so essentially marry many small decentralized communities on a searchable platform. Anonymous but karma/identity weighted profiles, curating and debating lists of their favorite things around a single topic.

Aggregate them into a single master index with nice UI, then pay out equitably based on performance.

There’s a lot of questions on preventing spam and fraud, and I think the further away you get from “high ticket products” the more you avoid that problem, but it can’t be worse than Google: in one you’re relying solely on magical AI and links, the other you can have as much magical AI and links as you want, but you also have user accounts which maybe invited each other and can vouch for each other at a cost, votes that can be weighed by all sorts of factors, and basically a much richer graph of self-moderating people to back the index.

Working on something like this at the moment!

>... simply acknowledging small communities curate the best content

This is true for some things. But for reliability reviews, for example, there's a certain power in large numbers. I'd much rather see reviews for common appliances from 200k people (or more) than the same from a group of 2000 appliance-centric folk. That's true even though the 2k set will likely know and care a lot more than many of the people in the 200k set.

On the other hand, with large numbers of people, it's much harder to deal with "95% of reviews are complaints" problem, which tends not to be so much of a problem with "enthusiasts".

Why not both? Rotten Tomatoes gives the critic score side by side with the audience score, as does Metacritic.

There's arguably value not just in both scores, but also both being presented side by side.

Rotten Tomatoes is the epitome of why trusting expert ratings on their own is a bad idea. The number of times that the tomatometer is wildly off the general perception is insane, enough to make the tomatometer largely useless for myself at lease.

It certainly does seem, though, that a company with Google's prowess in search, NLP, ML, algorithms, categorisation and the like should be able to do a lot better, even if the problem is incredibly difficult.

My only explanation is that it doesn't suit the business needs of Google to do so.

Part of the spam is clearly condoned by Google, probably for a mix of reasons.

An example is Pinterest, Google can drop or deprioritize their properties, like they do with other actors. But for some reason they do not. Maybe they are scared of litigation.

> acknowledging small communities curate the best content

Agreed. The problem is that Facebook and Google more or less successfully disintermediated those small communities of old, namely newspapers, magazines, websites based on region or interest which had a vested interest in their constituency. In doing so, the tech giants swallowed up the advertisement dollars going to those old media, and made a few people insanely rich while destroying the entire ecosystem.

> Does anyone think that this can possibly be at the core of Google's page rank anymore?

Yes, it is, and it still makes up the super majority of factors. It's the reason why people buy expired domains (that have links pointing to them), it's the reason why they pay lots of money to rent a subdomain or folder on cnn.com. It's the links, and it's only the links. And it works great.

The actual content on the site has little to do with the ranking. Yeah, sure, you need to have the keywords in there, but you don't need quality content, you just need quality links.

So you're saying that the page rank for http://foo.bar.com/baz.html is more a function of the link score for foorbar.com than the link score for baz.html or its contents?

No, but you do have internal links on your site as well, and they work just like external links, passing on "link juice". If you have a million important links pointing to / and zero external links pointing to /baz.html and you link from / to /baz.html, you'll still see /baz.html hike up the SERPs.

https://news.ycombinator.com/item?id=27991322 this submission from today is exactly the same. Expired domains are the big business these days, and they can absolutely catapult you into the top 3 without any change in content, performance or layout.

This seems quite contrary to Brin & Page's original explanation of their (first) page rank algorithms, which was boosted as utilizing other people's linkage as the justification for a higher ranking. It also seems ripe (ruined, even!) for being completely weaponized. Another failure of the imagination, maybe.

It makes sense though, I believe. If you link to the home page of a website, what would that mean if you consider it a vote: "I endorse this precise document", or "I endorse what's on that website"? It would make sense to lessen the endorsement if you link to individual pages only, but not the home page / hub pages.

SEOs have, of course, gone to extreme lengths with that by doing "pagerank sculpting", i.e. making sure you mostly link to the pages you want to rank well with, and not your privacy policy. From my experience, they've stopped being overly sensitive about that now ("it has to look natural"), and it's just about raw power: who can buy the best expired domain (state entities being dumb and just letting their domains expired when they rename...), or who can rent a folder (and the links that come with it) on the strongest media platform.

It's a cat-and-mouse game, and google seems to not want to be a cat unless they can be a perfectly automated cat, and they can't.

I see reviews of best X under $Y that I would have some trust in their conclusions on websites that also do news about the subject area. I don't see why these websites should not be able to use affiliate links.

Some examples: Cycling Weekly for bicycles and AnandTech for computing stuff.

I don't go to their websites via Google search though.