Hacker News Reader

Comment by mceoin | Hacker News Reader

Excerpt worth pulling out for HN readers:

Crawling incidentally I think is the biggest issue with making a new search engine these days. Websites flat out refuse to support any crawler than Google, and cloudflare and other protection services and CDN’s flat out deny access to incumbents. It is not a level playing field. I would actually like to see some sort of communal web crawl supported by all web crawlers that allows open access to everyone. The benefits to websites would be immense as well, as they could be hit by a single crawler, rather than multiple, and bugs could be ironed out.

View on HN

While it's an obstacle, I don't think this is as big of an issue as they make it out to be, and I say this as someone who runs a DIY crawler and a janky homebrew search engine. You can get listed as a good bot with cloudflare &c if you ask nicely enough. Even if you aren't, their IP ranges are public so you can crawl them really slowly if you absolutely must have their stuff.

In the end, you are never going to get completeness, that's a bad mark to aim for, and not even Google does that. That goal requires you have a model of the Internet that's as old as Vannevar Bush. The Internet isn't a bunch of static files on a souped up FTP server anymore, most pages are generated request-time, increasingly client-side. There are websites that don't just have infinite documents, they have infinite wildcard subdomains as well. There are websites that give you different results depending on what time of day it is, or where you are from.

The hard problem of search engines is to find a good subset of the internet for some definition of good; and the first step toward doing that (in a timely fashion) is to rein in the crawling. Search engines are always always going to present a selection of web sites, and there are always going to be pages you can't find because they didn't make the cut; either because they were excluded, or because they excluded themselves.

Hey, your project is awesome, but you seem to be the only non-big3 crawler who doesn't encounter this problem and call attention to it [1]. Maybe because you (awesomely) focus on low/no-javascript, text-heavy sites? Much of the most obnoxious antibot stuff (reCAPTCHA, CDN "AI" throttling, a lot of what cloudflare does, etc) is javascript-based.

Maybe I'm totally wrong here. If so, you really should make your crawl corpus publicly available. Trickle it into one of the public clouds and charge people a few microcents per query. You'll get to retire early, knowing you've done a good deed for human technological progress.

> There are websites that give you different results depending on what time of day it is, or where you are from.

That has been true since at least 1993 [2], before there were search engines!

> The hard problem of search engines is to find a good subset of the internet for some definition of good

This really is a case of the tech industry trying to redefine "search" in a way that gives them editorial power. Librarians don't do that sort of thing. We could learn a lot from their integrity. Or we could just stop calling it "search", because that's not what it is anymore. For example, that box at the top of amazon.com is most definitely not a search box anymore; it's a sales-lead-generator box. The button should be marked "try to sell me shit which is not necessarily related to these words" instead of "search".

[1] off the top of my head: archive.is/archive.today, gigablast. I'm actually having trouble remembering any still-operating whole-web crawlers besides Google, Bing, and Yandex.

[2] https://en.wikipedia.org/wiki/Common_Gateway_Interface

Some folks think it's a very big issue and have released data on their findings about robots.txt files: https://knuckleheads.club/the-evidence-we-found-so-far/

We see some problems but as an established, and respectful crawler, we are usually able to overcome most blocks at Mojeek; a international web search engine with no tracking (currently 4.3 billion pages)

I learned of Mojeek via searX use, it has found corners of the Internet which the big 3 have ignored with the same search queries; I find it very useful to balance out all the popular results. Kudos to your team.

I'm aware that CGI has been around for a long time, but it was bog slow since it forked new processes to deal with every call, it wasn't really feasible to build the sort of dynamic websites we do today until stuff like PHP entered the market.

The relative proportion of static files versus dynamic web content have probably reversed.

> This really is a case of the tech industry trying to redefine "search" in a way that gives them editorial power.

This isn't a new development at all. The signal to noise ratio problem goes back to the late '90s, and ruthlessly removing noise was arguably one of the moves that allowed Google to break through.

Unfortunately they seem to have forgotten why they became a success.

From http://infolab.stanford.edu/~backrub/google.html

> [...] In 1994, some people believed that a complete search index would make it possible to find anything easily. [...] However, the Web of 1997 is quite different. Anyone who has used a search engine recently, can readily testify that the completeness of the index is not the only factor in the quality of search results.

It's difficult to conceptualize these numbers, but we can use English Wikipedia as a measuring stick. It has of the order ten million pages, and covers most topics you would ever want to read about. Maybe there are some niche things here and there that are missing, but overall, it's a relatively complete coverage of human ideas and interests. How many English Wikipedia Units do you need, for your search engine to feel complete, assuming 100% signal and 0% noise? Do you need one? Ten? A hundred?

A "big search engine" crawl is over a hundred thousand English Wikipedia Units. How much of that is interesting? How much has been written by humans, has ever been looked at by humans?

https://commoncrawl.org/ is a big crawler with all data made public. Probably a good resource for new search engines.

Yes and no. It works for the deep web, but for anything that needs more frequent crawling no. They don't do it often enough.

I have mentioned this elsewhere, but id love there to be a single crawl that everyone was able use and build indexes off. It would be especially good for webmasters too as you would have one bot to worry about and optimise for, reducing the load on systems, and allowing for innovation in the search space that cloud flare and other cdn's are stifling because they understandably are blocking crawlers that aren't Google or Bing.

For example id love to see something added to robots.txt that allows you to say "crawl at 2am, hit has hard as you want, don't come back for a week" or some such. Having a single crawl could allow for this.

Your comment has been misplaced, it should be under https://news.ycombinator.com/item?id=28665395 (I was pretty confused until that realization).

Well, then change your user agent to 'googlebot' when crawling the web, since it now became the synonym for search crawlers?

That trick doesn't work because people also filter out Google's IP addresses to verify that the "googlebot" crawler is actually a real Google bot.

Can you run a crawler from GCP or do they explicitly filter for other Google AS?

Reply on news.ycombinator.com