Hacker News Reader

Comment by hak8or | Hacker News Reader

For those of us datahoarders, how large is sci hub? Surely it compresses extremely well considering it's all human readable text? I would love to mirror it via a torrent or on ipfs and whatnot, and I have a gigabit connection with roughly 80TB of space I can throw at this.

View on HN

It's mostly composed of PDFs actually. Judging from my own library the average publication is about 2 MB, and I believe about 1e8 publications are currently indexed, so it's on the order of hundreds of TB.

Either a) those pdfs contain images or other graphics that don't compress easily or b) the text is not compressed.

There are definitely many images and graphics. I don't know if my sample is representative of most literature, but I don't think it's too far off..

Or c) The text is terribly compressed. Probably a lot of the "text" is also a scan of text as well.

Could be an interesting project to try and make a sci-hub optimized compression system.

DejaVu format

This got me curious. For reference, my Zotero library is predominantly biomedical in nature. This device currently has 810 of the pdf files pulled down to the local drive, of which the vast majority are journal articles (a very few of the largest ones are presentation slide decks).

  Size Range   Count
  [0,   0.5)   255    (31%)
  [0.5, 1)     179    (22%)
  [1,   1.5)   108    (13%)
  [1.5, 2)     69     (9%)
  [2,   3)     80     (10%)
  [3,   10)    95     (12%)
  > 10         24     (3%)

That's ~50% < 1 MB and ~25% > 2 MB. I'm actually a bit surprised they're that small.

btw anyone knows if there are PDF shrinkers ?

some journals can weight a lot (scanned pages) even though there's potential to recompress images while keeping it readable

I've used Ghostscript to shrink scanned PDFs for emailing, although I don't know how lossy the compression is.

[0] https://askubuntu.com/a/256449

I tried it but it was cpu intensive for very small improvements

Much (but I believe not all? I'd have to look more into it) is backed up on libgen: http://gen.lib.rus.ec/repository_torrent/

There's also the very bottom file here: http://download.library1.org/dbdumps/

Are SciHub torrents not here: http://gen.lib.rus.ec/scimag/repository_torrent/

I think your link may point to the LibGen torrents (books, mainly, I believe).

Yep, you're right!

I looked at a few of the torrents, and they're very roughly 150 GB each. So that's about 50 TB overall.

More like 70T.

Thanks. I was too lazy to look at them all.

Currently 70T growing at about 150G every two weeks or so.

That means if you have access to a LTO-6 tape drive (best price per TB right now at around 8,88€ per TB) it only costs around 622€ ($700) to create a backup.

Reply on news.ycombinator.com