How We Built Our 60-Node (Almost) Distributed Web Crawler

OP here. This post is an expansion of my comment http://hackerne.ws/item?id=4369590 on the excellent post 'How to crawl a quarter billion webpages in 40 hours' ( http://hackerne.ws/item?id=4367933)

Don't hesitate to ask me any questions.