Comment by technokrat233 | Hacker News Reader

Comment by technokrat233 | Hacker News Reader

Parent Full thread

Having worked with/on two large-scale web crawling systems at FAST Search & Transfer; The C-Crawler that powered alltheweb.com (owned now by Yahoo) and a python-based crawler that was a workhorse for FAST ESP/Scirus.com/etc.. (now owned by Microsoft); this run-down is pretty good. Some things I'd add:

- Link traps. If you limit to 100 pages per site not as big an issue but if you want to go deeper you need a way to detect when a site is generating garbage.

- Near duplicate detection. There's lots of sites like you mention that republish content of others, but some just present it in different ways with different headers, timestamps, etc..

- Content/meta-data detection/extraction, once crawled you want to do something with the content and detecting the actual content of pages is non-trivial if you don't want headers/ads/etc..

- How do you handle non-HTML content (PDF, Docs, etc?)

- How do you handle large content (sample, truncate, ignore)?

I used to work at a vertically-focused web search engine and ran the operational side of the crawler.

Also missing from this discussion would be a mechanism to rate limit (and determine adequate rate limits, based on your error rates) the crawl.

Also, detecting that you've been blocked and backing off so as not to further hammer the site you're crawling with requests. Related:

IP management is an issue here as well: lots of places just carte blanche block whole ranges from crawling activity. And will you be honoring robots.txt or not?

Be prepared for people to block you in new and stupid ways: once got blocked from hitting the site's name servers to even do lookups against them. They blackholed our packets. So what should have been a ~500ms DNS query at each http request turned into a 15s pause while the DNS request timed out ... eventually this stacked up across all threads, backing the overall crawling infrastructure to deadlock.

The Wayback Machine architecture is probably a good, public implementation of a large scale crawling mechanism. This post[1] about it may be a bit dated, but it's probably still accurate.

[1] http://highscalability.com/blog/2014/5/19/a-short-on-how-the...

Reply on news.ycombinator.com