Ask HN: Crawl the entire web?

2 pointsnullundefined1 comment

If you were tasked with crawling the entire web, how would you go about doing it?

I'd grab the CommonCrawl data and keep moving on what my actual problem is.