Comment by bogidon | Hacker News Reader

Comment by bogidon | Hacker News Reader

I feel like all that's ever talked about here as bad about search is privacy. But what about good results? Google search results have an extremely heavy commercial bias. What would the internet look like without that? I would really like to have a say in how the ranking algorithm works. If I could, for example, exclude results from the Alexa Top 100k with a toggle, I bet I'd find interesting content a lot easier.

Most "privacy-centric" search engines I've seen – especially DDG – do not approach this, probably since they use Bing results, which is just as inflexible and SEO-able as Google. Has anyone had a good experience with a search engine that implements a novel crawler and/or ranking algorithm?

I've been trying Yandex for some searches lately, and it's results seem a little less commercially catered, but there are indeed some serious privacy concerns with it [0].

I have been holding a belief lately that a nonprofit search engine that scrapes & ranks in a way that is not commercially-driven is the single most important missing piece of digital infrastructure... anyone share this view?

[0] https://nelsonslog.wordpress.com/2020/01/07/facial-recogniti...

Changing the search provider wouldn't matter. The commercial nature of Google's results has almost nothing to do with Google and almost everything to do with SEO. Search results are far too valuable a piece of real estate for rankings to not be correlated with money. I guarantee you the army of marketers and full time SEO ninjas at Conde Nast will always figure out a way to get their content past your "no corporations allowed" filter.

I'm advocating for a ranking algorithm aligned to the benefit to the users, instead of the marketers. I think you're not appreciating Google's ranking algorithms lending a hand to the popularity of SEO. Their rankings are certainly not objective.

As for "far too valuable of a piece of real estate"... I feel like that's an unimaginative view trapped in the commercial state that we're in. Perhaps if Google had beat Wikipedia to online encyclopedias, but had pumped theirs with ads, we'd be saying the same thing about the far too valuable real estate of online information.

I don't think that an algorithm "designed to the benefit of the users" is nearly as straightforward as you think it is. We didn't get into the current situation because Google deliberately chose to make their algorithm friendly to marketers; we got here because marketers competed with each other to rank up in search results, effectively reverse-engineering Google's algorithm in an effort to exploit it. This would happen to any search algorithm that was widely adopted enough that marketers started trying to optimise for it.

(Google has deliberately slanted their news results towards mainstream sources in an attempt to suppress political messaging that they consider unacceptable, but it's not clear whether this is having an impact on the specific problem of commercial results drowning out other sources.)

> We didn't get into the current situation because Google deliberately chose to make their algorithm friendly to marketers.

This isn't quite true. Google has changed its search results in recent years to cut out content from more than a decade or so ago. You can no longer find the content even if you search for the exact strings in the website. If you have your own old site, it is easy to see from the logs that the Google Bot is still crawling the old pages, it just doesn't index them any more.

There are many instances where the most relevant content to return for a search would be from a website from the early millennium, but Google refuses to show that, while Google will show a big commercial website from more recent times even if the result is less relevant to the query.

Good point; this is in addition to their other missteps and contributes to the problem. The early web was less commercial, so phasing out old results pushes ever harder for commercial results over others.

Why doesn't Million Short suit you? It is precisely that toggle to cut out the big commercial websites.

I didn't know about it, so big thanks. That for sure will be a useful tool. However, that was only one example of what I'd like to be able to do. I'd also really like visibility into the ranking algorithm and to be able to tweak the criteria it prioritizes to suit the interests of my search.

Reply on news.ycombinator.com