Hacker News Reader

Comment by lgierth | Hacker News Reader

I added the raw .gz files to IPFS when Library of Congress announced this last week: https://github.com/ipfs/archives/issues/152

It's slightly more than 100GB and here it is: https://ipfs.io/ipfs/QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtH...

(Note that the filesizes in the directory listing are all wrong -- that's the original index.html from loc.gov/cds/downloads/MDSConnect/)

This makes it a lot easier to use this dataset at e.g. hackathons, where a lot of people would simultaneously pester that LoC server, which already seemed pretty bandwidth-limited on its own when I downloaded the files.

View on HN

You can pin it: `ipfs pin add QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9`

Or list it: `ipfs ls QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9`

Or copy it into your local filesystem `ipfs get QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9`

Did you push them into the Internet Archive yet? If not, going to grab a beer and start iterating through your IPFS objects.

Go for it! :) I've never pushed anything to IA so far to be honest.

For future reference then! https://github.com/jjjake/internetarchive

If someone wants to pay for / donate 100GB worth of bandwidth on a VPS somewhere, I'll do it.

scaleway: €2.99/m 200Mbit/s Unmetered bandwidth

Okay, so who's going to spot me 3 EUR? ;)

Could TransferWise you the $3.35USD. That work?

Having never done this before I've got to have a look at the internetarchive tool first, but yes, that would work (I'd hate to take your money then not be able to deliver).

This thread is getting too long; how do I contact you?

This amount of usage falls well within most VPS companies' free trial / promo codes offerings and should cost you nothing. Use a throwaway email account and drop it after a month.

AWS will give you a whole year if you haven't tried them yet and the other popular VPS companies (DO, Linode, etc.) all will give you at least $10 startup credit. This is probably simpler and faster than figuring out how to receive <$4 from some random internet commenter.

Yes. This isn't the first time I've tried to contact toomuchtodo.

Anywho, the amount of data is actually ~19GB which is well within what I can upload with my home connection. Unfortunately the ia tool is failing for me: https://github.com/jjjake/internetarchive/issues/176

Also, it's not really about the $3, more that a tonne of "$3" projects really add up over a year or so.

Email sent.

I have been advised that the data already exists on archive.org.

Email me back. Would still like to buy you a beer for your troubles.

Oops, it turns out I somehow ended up with decompressed files, that's why the filesizes are so far off. I'm creating a new fixed dump.

The correct dataset (with gzip compression) is here: https://ipfs.io/ipfs/QmcvfB6pAqUfTnuAK8zFKVxbdhopnBPveJrDcy1...

Reply on news.ycombinator.com