How to crawl a quarter billion webpages in 40 hours (2012)

Another possible approach is for sites to be self-indexing, with sharable indices, and validation (and penalties) for deceptive practices (either false term inclusion or exclusion).

The simple existence of a websearch and index protocol would wipe virtually all of Google's present value.

I came here to state just this. What would code for the crawler in the article do that premade DDoS bots don't already do? (already does that task better too) Even in 2012 when the article was published, there were plenty of open source crawlers AND tools designed specifically for "burdening websites with traffic" that were available.

The only problem is: nobody would care, people will still use Google because it's all they know. Some people even ignore the fact that underneath they use an operating system and a browser.

That 10x average seems to be a bit off considering our data, which is of course spotty since it's crawled by a third party.

But to give some numbers, in one of our experiments we filtered web sites from the archive for known entities and got 307,426,990 unique URLs that contained at least two of those entities (625,830,566 non unique) and in there were only 5,331,272 unique hosts. That archive contains roughly 3 billion crawled files (containing not only HTML, but also other MIME types) and covers mostly the German web over a few years.

There are a lot of hosts that have millions of pages. To name a few: Amazon, Wordpress, Ebay, all kinds of forums, banks even. For instance, www.postbank.de has over a million pages and they were not re-crawled nearly that often.

> It must be noted that around 75% of websites today are not active, but parked domains or similar.

So actually more like 0.5B websites. Feels quite tiny. Seems most activity online really is behind walled gardens like FB.

Try crawling ONE wordpress blog with less than 10 posts and you will be surprised just how many pages there are due to pagination on different filters and sorting options, feeds, media pages, etc.

I think the cost of fetching and parsing the data is much less than the cost of building an index and an engine to execute queries against that massive index.

https://stackoverflow.com/questions/5834808/designing-a-web-...

You can do a lot of other things too, but the above already takes most of the sting out of such traps unless you use a huge number of domains.

Sure. In the simplest case you just look at the User-Agent header. Your browser will send one thing, and something like GoogleBot will send something else. Now, if you're a website owner who wants to block bots, you can't just depend on that, because somebody writing a bot can trivially put any string in that header that they want. But there's a lot of other ways you can tell. A simple way to discriminate somebody who's just using curl or wget, for example, is to serve a page with some javascript in it and check if the javascript is executed or not. Usually you'd do something like this from a proxy that sits in front of your actual content, and throw out subsequent requests from a UA that fails the check. Of course identifying the UA consistently is yet another challenge.. if the thing handles cookies properly, you can use a cookie. You could try going by IP, that that's dicey in various ways. Etc., etc., yada, yada.

All in all, there's a constant arms race going on between the companies that want to block bots / crawlers, and the people who want to crawl/scrape content. The techniques on both sides are constantly evolving.