Hacker News Reader

Comment by radus | Hacker News Reader

It's mostly composed of PDFs actually. Judging from my own library the average publication is about 2 MB, and I believe about 1e8 publications are currently indexed, so it's on the order of hundreds of TB.

View on HN

This got me curious. For reference, my Zotero library is predominantly biomedical in nature. This device currently has 810 of the pdf files pulled down to the local drive, of which the vast majority are journal articles (a very few of the largest ones are presentation slide decks).

  Size Range   Count
  [0,   0.5)   255    (31%)
  [0.5, 1)     179    (22%)
  [1,   1.5)   108    (13%)
  [1.5, 2)     69     (9%)
  [2,   3)     80     (10%)
  [3,   10)    95     (12%)
  > 10         24     (3%)

That's ~50% < 1 MB and ~25% > 2 MB. I'm actually a bit surprised they're that small.

Either a) those pdfs contain images or other graphics that don't compress easily or b) the text is not compressed.

There are definitely many images and graphics. I don't know if my sample is representative of most literature, but I don't think it's too far off..

Or c) The text is terribly compressed. Probably a lot of the "text" is also a scan of text as well.

Could be an interesting project to try and make a sci-hub optimized compression system.

DejaVu format

btw anyone knows if there are PDF shrinkers ?

some journals can weight a lot (scanned pages) even though there's potential to recompress images while keeping it readable

I've used Ghostscript to shrink scanned PDFs for emailing, although I don't know how lossy the compression is.

[0] https://askubuntu.com/a/256449

I tried it but it was cpu intensive for very small improvements

Reply on news.ycombinator.com