Triv.io donates URL index to Common Crawl | Hacker News Reader

Triv.io donates URL index to Common Crawl

commoncrawl.org

51 pointsLisaG16 comments

Is it just me or is the data file not available for free despite being in the Amazon Public Dataset S3 bucket?

*Edit: The problem seems to be fixed now.

Let me double check that for you. Were you using a valid aws-id and secret?

No, usually you can download them without sending any aws-id if they are on the Public Datasets S3 bucket, e.g.

    wget https://s3.amazonaws.com/aws-publicdatasets/common-crawl/crawl-001/2008/06/19/0/1213886083018_0.arc.gz

What I'd love to see: A simple list of domains No information about content, no full URLs, just the domainname.

Extracing such a list from the generated index only takes a small script and a few hours to download the 200+ GB index file. Which is a lot less than the slightly bigger script and the months/years to download and process 80+ TB of arc files that it previously would have taken to extract all domains.

Anyways, if you want a copy of domains from the index file just send me a mail to the address in my profile.

Where is the index file located in the s3 bucket?

s3://aws-publicdatasets/common-crawl/projects/url-index/url-index.1356128792

thank you kindly!

That would be great. Out of curiosity, mind describing what you'd use such a list for?

Usually as a starting point for a crawler when checking statistics about the web.

Crawling is much faster if you don't have to "spider" all the links and check if you've already visited them or not.

With a big enough list, you can just iterate over those domains. (average number of links on a website, how often does javascript framework x vs y get used, how many sites have an HTML5 doctype yet, ...)

You can get the top million sites from Quantcast in a download for free. Alexa used to also offer something similar, but I don't see it on their site any longer (they try to sell their data through aws, so that's probably why).

Can someone dig that up and link it please

http://ak.quantcast.com/quantcast-top-million.zip

You can find the Alexa top 1 million domain list at http://www.alexa.com/topsites (it's on the right hand side).

A number of the sponsoring orgs will grant you access to their zonefiles if you're willing to do a little paperwork (HelloFax is awesome) and have a valid reason for access. Granted it's not the complete list of registered domains (those without nameservers on file won't show up), but it's pretty close.

The list of sponsoring orgs for each TLD is at: http://www.iana.org/domains/root/db

Nice work triv.io!

Reply on news.ycombinator.com