Hacker News Reader

Comment by nneonneo | Hacker News Reader

The total collection is 54.54 TiB, with 690 torrents as of writing. For preservation purposes, I put magnet links to all the torrents here: https://pastebin.com/zTAqS7wz

So, even if the torrents go away, the magnet links should still be usable.

(Edited: fixed link)

View on HN

What I find more amazing than the fact that some people have been so neighbourly as to share such a volume of information, is that this is still just a tiny fraction of all human knowledge. Scientific journals are certainly an important collection, but still miniscule in comparison to all the other books out there. I have collected ~20GB of automotive service manuals and related data, and that's a tiny amount too. To speak nothing of the many terabytes of entertainment others have...

I've seen some Plex servers hosting in the amount of petabytes of content, it's crazy.

Downloading all the torrent files:

  mkdir torrents
  for i in {000..689}; do curl -sS "http://gen.lib.rus.ec/scimag/repository_torrent/sm_${i}00000-${i}99999.torrent" -o "torrents/sm_${i}00000-${i}99999.torrent" -m 30; done

...which themselves take up about 75 MB, so here is the full collection of torrent files for your convenience:

https://lab.brainonfire.net/drop/delete-after/20180630/torre... (please use the next link if possible, though!)

And a torrent that is basically equivalent to that file, thanks to the below commenter:

https://lab.brainonfire.net/drop/delete-after/20180630/scihu...

Each torrent appears to index 100 zip files (about 800 MB each), each of which presumably contains 1000... journal articles? I don't know. The seed isn't blazing fast, so it will be a bit before I can pull out a random zip file from this randomly selected torrent and inspect it.

I base64'd a torrent file containing the HTML index and all 690 current torrent files and stuck it here: https://pastebin.com/p1nN6veK. Would appreciate if you could upload that in a binary format somewhere.

Magnet link for simplicity:

  magnet:?xt=urn%3Abtih%3A97fc8218775e3a1e5b90607435c0581f839e0f1d&dn=repository_torrent&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.open-internet.nl%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.skyts.net%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.piratepublic.com%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2710%2Fannounce&tr=udp%3A%2F%2Fpublic.popcorn-tracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker1.wasabii.com.tw%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.zer0day.to%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.xku.tv%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.vanitycore.co%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.mg64.net%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.tiny-vps.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce&tr=udp%3A%2F%2Foscar.reyesleon.xyz%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.stealth.si%3A80%2Fannounce&tr=udp%3A%2F%2Fopen.facedatabg.net%3A6969%2Fannounce&tr=udp%3A%2F%2Fmgtracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fipv4.tracker.harry.lu%3A80%2Fannounce&tr=udp%3A%2F%2Finferno.demonoid.pw%3A3418%2Fannounce

I am currently seeding this small file and will do so for as long as I am able.

(Edit: updated magnet link and torrent to include trackers)

Thanks! Uploaded as binary to https://lab.brainonfire.net/drop/delete-after/20180630/scihu... and currently seeding.

I can't manage to load this magnet link in transmission:

  Error adding "magnet:?xt=urn%3Abtih%3A97fc8218775e3a1e5b90607435c0581f839e0f1d&dn=repository_torrent&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.open-internet.nl%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.skyts.net%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.piratepublic.com%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2710%2Fannounce&tr=udp%3A%2F%2Fpublic.popcorn-tracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker1.wasabii.com.tw%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.zer0day.to%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.xku.tv%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.vanitycore.co%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.mg64.net%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.tiny-vps.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce&tr=udp%3A%2F%2Foscar.reyesleon.xyz%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.stealth.si%3A80%2Fannounce&tr=udp%3A%2F%2Fopen.facedatabg.net%3A6969%2Fannounce&tr=udp%3A%2F%2Fmgtracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fipv4.tracker.harry.lu%3A80%2Fannounce&tr=udp%3A%2F%2Finferno.demonoid.pw%3A3418%2Fannounce": invalid or corrupt torrent file

Sorry, just tried this and apparently Transmission isn't a fan of the encoded colons (despite that being very valid URL syntax).

Try this one instead:

  magnet:?xt=urn:btih:97fc8218775e3a1e5b90607435c0581f839e0f1d&dn=repository_torrent&tr=udp:%2F%2Ftracker.coppersurfer.tk:6969%2Fannounce&tr=udp:%2F%2Ftracker.open-internet.nl:6969%2Fannounce&tr=udp:%2F%2Ftracker.skyts.net:6969%2Fannounce&tr=udp:%2F%2Ftracker.piratepublic.com:1337%2Fannounce&tr=udp:%2F%2Ftracker.opentrackr.org:1337%2Fannounce&tr=udp:%2F%2F9.rarbg.to:2710%2Fannounce&tr=udp:%2F%2Fpublic.popcorn-tracker.org:6969%2Fannounce&tr=udp:%2F%2Ftracker1.wasabii.com.tw:6969%2Fannounce&tr=udp:%2F%2Ftracker.zer0day.to:1337%2Fannounce&tr=udp:%2F%2Ftracker.xku.tv:6969%2Fannounce&tr=udp:%2F%2Ftracker.vanitycore.co:6969%2Fannounce&tr=udp:%2F%2Ftracker.mg64.net:6969%2Fannounce&tr=udp:%2F%2Ftracker.tiny-vps.com:6969%2Fannounce&tr=udp:%2F%2Fp4p.arenabg.com:1337%2Fannounce&tr=udp:%2F%2Foscar.reyesleon.xyz:6969%2Fannounce&tr=udp:%2F%2Fopen.stealth.si:80%2Fannounce&tr=udp:%2F%2Fopen.facedatabg.net:6969%2Fannounce&tr=udp:%2F%2Fmgtracker.org:6969%2Fannounce&tr=udp:%2F%2Fipv4.tracker.harry.lu:80%2Fannounce&tr=udp:%2F%2Finferno.demonoid.pw:3418%2Fannounce

I'll try to remember to do it when I get home, but if you beat me to it can you please check and see if that's a known issue with Transmission? Because that seems easy to fix, and in my mind it's bad news to need "vendor specific" magnet links

It doesn't look like they have an open issue for that. If you file an issue, you may wish to crosslink it with https://github.com/transmission/transmission/issues/249 which would probably be solved by the same fix for URL query handling.

It sounds like their URL handling might just be terribly broken; see for example https://github.com/transmission/transmission/issues/608.

[wget] wget -r -l1 -H -t1 -nd -N -np -A.torrent -erobots=off http://gen.lib.rus.ec/scimag/repository_torrent/

Is there a way to see which are in the direst need of seeders?

I am currently participating in some of them at random, as I have nowhere near 50 TiB of free space but who love to do it more efficiently.

Possible in theory, just ping the trackers with each of the torrents and see which ones have the fewest peers.

Even if an entire satellite transponder (36 MHz) was used to broadcast these files (assuming 100Mbps broadcast/downlink), it would still take two months to download the whole thing.

Well, that depends on the code rate and modulation.

DVB-S2X 32APSK 32/45 could do 150Mbps with the same bandwidth (as in, the same amount of radio space, not internet bandwidth).

???? This is obviously equal to 54,000 datasets of 1 gigabytes each, which I would say is a pretty large dataset to publish along with a paper, so it's hard for me to even imagine 54 thousand such 1-gigabyte datasets. A one gigabyte PDF is astronomical in size, PDF's are usually far far shorter.

So why is this so large? Aren't they just PDF's, basically? And usually just a few pages? Excuse my ignorance. I'm very surprised at the size that you quote (54 TiB).

There’s dozens of millions of articles total, not thousands.

wow. I didn't know there were a grand total of dozens of millions of journal articles published total worldwide. My guess would have been hundreds of thousands at most. These are journal articles, not high school book reports, right? I'm astounded at the number you quote.

For example there are about 10,000 papers submitted per month to all of arxiv[1] - which is a huge number. That totals 132k per year or so. For there to be "dozens of millions" of journal articles, that would mean arxiv has just 0.83% of them. Given the very low bar to publishing on arXiv I would think if you compared the number of arxiv articles as a percentage of all pubished journal articles, it would be a lot more than 0.83%! So the "dozens of millions" of journal articles is extremely astounding to me.

https://arxiv.org/stats/monthly_submissions

arXiv only handles a small corner of scientific publications, so 0.83% doesn't sound out of whack. Most folks in biology or chemistry aren't on there, for example.

A Nature study from 2014 (http://blogs.nature.com/news/2014/05/global-scientific-outpu...) pegs the number of papers published between 1980 and 2012, with at least one citation, at 38 million. So, 69 million publications including uncited ones and ones from outside this time range actually sounds like an underestimate - that is, the 69 million paper archive is probably missing a fair number of articles.

There are a lot of scientists and they work hard (and get paid peanuts, which seems unfair).

Unfair and utterly stupid of humanity.

54TiB for publications only ? do you know the total for the books too (which is more libgen than scihub). I remember a 200PiB estimate but I'd love to know more.

Do you recommend a seedbox for this? I'm currently with Ultraseedbox which is only 3TB. I am thinking of getting a new service specifically to download then provide another source/host of these files.

Reply on news.ycombinator.com