Hacker News Reader

Comment by soult | Hacker News Reader

Is it just me or is the data file not available for free despite being in the Amazon Public Dataset S3 bucket?

*Edit: The problem seems to be fixed now.

Let me double check that for you. Were you using a valid aws-id and secret?

No, usually you can download them without sending any aws-id if they are on the Public Datasets S3 bucket, e.g.

    wget https://s3.amazonaws.com/aws-publicdatasets/common-crawl/crawl-001/2008/06/19/0/1213886083018_0.arc.gz