Hacker News Reader

Comment by exDM69 | Hacker News Reader

> I for one would really like a say 3 GiB version of English Wikipedia without Pokemon and Porn, or a 33GiB version with useful images.

There are several offline snapshots of different sizes of Wikipedia that do exactly this. Some of them are more thoroughly curated, others are a snapshot of the most popular articles, etc.

What I could not find easily was a full English Wikipedia with images and all available in an easily accessible format (like the Kiwix snapshots). I'd put a whole hard disk for this purpose, because the size is in the gigabyte-terabyte range.

View on HN

Some relevant links:

100,000 curated pages:

https://en.wikipedia.org/wiki/Wikipedia:Vital_articles/Level...

In terms of space, the text is over 1000x smaller than the multimedia: ~20GB (uncompressed) for the text vs ~45TB (compressed) for multimedia (est accounting for growth vs 23TB@2014).

https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia

I am aware that there are various offline wikipedia compilations and I am grateful they exist and to the people who work on them, but last I looked using them looked like a bit of pain, and on checking again that still appears to be the case. I get the impression that if I want to have something of decent quality that fits into the 10s of GiB range I'd consider reasonable¹, I'd have to manually munge together several non-Pokemon topics I consider important plus one of the text only dumps. This would be more palatable if I had a strong impression that once I hit upon a working process I could rinse-and-repeat on a yearly basis.

Do you have any experience with this?

¹E.g. the 100k top article page linked by sibling does not look super promising, ~700 math articles vs ~10k fluff (sports, entertainment etc.).

I have the ~80GB "full english" snapshot I grabbed some time ago and use with Kiwix. It's not quite "full", it doesn't have quite all images and video, etc. But the size is still reasonable for a desktop/laptop PC.

I also have a 10GB snapshot on my phone, but it's a bit lacking. I don't have enough storage on my phone to fit the bigger one. But a battery powered offline Wikipedia is high on my list of necessary things in case things go south.

I haven't found a great way to keep these synced up, glad to hear if someone has a good solution for up-to-date offline wikipedia.

What's wrong with kiwix snapshot of English wikipedia? It's only 93GB.

http://download.kiwix.org/zim/wikipedia/wikipedia_en_all_max...

Well, I was hoping for less than 93GiB compressed, given that this exceeds the storage capacity of several devices I'd like to use it on and I'd estimate the total amount of info I care about to be far smaller than that. And small is beautiful: if everything fits into ram, you can do various interesting data munging tasks on the data (which is after all not uninteresting) very quickly.

Reply on news.ycombinator.com