Ask HN: What Happened to the Cuil Crawl Data?

4 pointsarayh2 comments

I was trying to look through the Cuil crawl data on archive.org but nothing comes up from the collection tab under any search query. I also can't seem to find anything that suggests that the Cuil crawl data was removed from archive.org. Any idea what happened here? Has anyone been able to access it recently? Maybe it's just temporarily down without any notice?

https://archive.org/details/cuilcrawl

View on HN

The items from the Cuil collection were "made dark" (i.e. you can not directly view/download them). I do not know the specifics, but as far as I know most web crawl data on the Internet Archive is not directly downloadable, but you can use the Wayback machine if you are looking for a copy of a specific website.

Depending on what you need the data for, Common Crawl[1] might be an alternative.

1: http://commoncrawl.org/

If you don't get an answer here, try asking Jason Scott @textfiles on Twitter. He's kind of the "face" of the Internet Archive and he's pretty good at directing queries like this to the right people.

Reply on news.ycombinator.com