Hacker News Reader

Comment by kenty | Hacker News Reader

Encarta was an amazing product that I fondly remember using in school. It was essentially wikipedia before there was wikipedia with the upside that it had curated and citable sources and generally a much higher quality of writing.

In my eyes, over the years, wikipedia has caught up a lot in terms of quality of writing and wikimedia now has a lot of images and video.

I wonder if there is a way to just download the whole wikipedia for easy distribution like encarta.

View on HN

Part of the value of Encarta was in the human curation of what was interesting.

Wikipedia includes (by necessity) everything — but an encyclopedia would probably devote more space to (say) Nigerian history than to (say) the in-game statistics of all ~1,000 Pokémon.

I suspect the sort of person who writes Wikipedia articles about Pokemon would not go and write articles about Nigerian history instead if Pokemon articles were not allowed, and vice versa. Wikipedia contributions are not really a zero sum game.

Well, for storage (and thus offline) purposes it is of course exactly a zero sum game. I for one would really like a say 3 GiB version of English Wikipedia without Pokemon and Porn, or a 33GiB version with useful images.

However broad and non-deltionist might be best for online Wikepedia, a more curated i.e. deletionist version would certainly be very useful for offline usage and quite a few other purposes.

> I for one would really like a say 3 GiB version of English Wikipedia without Pokemon and Porn, or a 33GiB version with useful images.

There are several offline snapshots of different sizes of Wikipedia that do exactly this. Some of them are more thoroughly curated, others are a snapshot of the most popular articles, etc.

What I could not find easily was a full English Wikipedia with images and all available in an easily accessible format (like the Kiwix snapshots). I'd put a whole hard disk for this purpose, because the size is in the gigabyte-terabyte range.

Some relevant links:

100,000 curated pages:

https://en.wikipedia.org/wiki/Wikipedia:Vital_articles/Level...

In terms of space, the text is over 1000x smaller than the multimedia: ~20GB (uncompressed) for the text vs ~45TB (compressed) for multimedia (est accounting for growth vs 23TB@2014).

https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia

What's wrong with kiwix snapshot of English wikipedia? It's only 93GB.

http://download.kiwix.org/zim/wikipedia/wikipedia_en_all_max...

Well, I was hoping for less than 93GiB compressed, given that this exceeds the storage capacity of several devices I'd like to use it on and I'd estimate the total amount of info I care about to be far smaller than that. And small is beautiful: if everything fits into ram, you can do various interesting data munging tasks on the data (which is after all not uninteresting) very quickly.

I am aware that there are various offline wikipedia compilations and I am grateful they exist and to the people who work on them, but last I looked using them looked like a bit of pain, and on checking again that still appears to be the case. I get the impression that if I want to have something of decent quality that fits into the 10s of GiB range I'd consider reasonable¹, I'd have to manually munge together several non-Pokemon topics I consider important plus one of the text only dumps. This would be more palatable if I had a strong impression that once I hit upon a working process I could rinse-and-repeat on a yearly basis.

Do you have any experience with this?

¹E.g. the 100k top article page linked by sibling does not look super promising, ~700 math articles vs ~10k fluff (sports, entertainment etc.).

I have the ~80GB "full english" snapshot I grabbed some time ago and use with Kiwix. It's not quite "full", it doesn't have quite all images and video, etc. But the size is still reasonable for a desktop/laptop PC.

I also have a 10GB snapshot on my phone, but it's a bit lacking. I don't have enough storage on my phone to fit the bigger one. But a battery powered offline Wikipedia is high on my list of necessary things in case things go south.

I haven't found a great way to keep these synced up, glad to hear if someone has a good solution for up-to-date offline wikipedia.

Meh. Even my phone can store the entire text of English Wikipedia and still have half the storage left over for sloppy OS and App code.

I think the point was that if you are looking at a subject where the relative importance is less obvious than Pokemon vs Nigerian politics then you could gauge importance implicitly from the existence of the article in a curated encyclopedia - not that contributers should be forced to spend more time on more important topics.

If they were employed to do so, they would. A curated encyclopedia with paid researchers would not be bound by the whims of weebs.

Some people call this phenomenon "wikigroaning": "the art of highlighting Wikipedia's bias toward things that don't matter".

https://en.wikipedia.org/wiki/Wikigroaning

https://www.somethingawful.com/news/wikigroaning/

I just don't get why doesn't it matter . for what it is Pokemon is a game and in some ways a form of art, and since many humans are interested in it deeply that means that it is important to them. I think it is important human knowledge, there is no need to be elitest about it.

Some pokemons are inspired in real world animals. Many people know the pokemon and not the animal.

They know about drowzee but not about the tapir.

And yet there's a ton of information that is rejected from Wikipedia itself, which found a place on e.g. Wikia and a few other sites like that - mainly about gaming, but also about subjects like D&D, etc.

If Bulbasaur is on Wikipedia, then Owlbear should be also. ;-)

https://en.wikipedia.org/wiki/Owlbear

I remember a similar comparison with Tony Blair back when he was the Prime Minister of the UK; the response was that while Wikipedia did indeed say more about Pikachu than Blair, Wikipedia also said more about Blair than did whichever “proper” encyclopaedia Wikipedia was being compared with at the time.

Well, Pikachu has been a highly influential Pokémon for nearly 30 years.

I recall reading that there was a time the Wikipedia article on the lightsaber was longer than the article on the printing press. No longer the case, thankfully.

Check out Kiwix: https://www.kiwix.org/

“Kiwix is an offline reader for online content like Wikipedia, Project Gutenberg, or TED Talks. It makes knowledge available to people with no or limited internet access. The software as well as the content is free to use for anyone.”

Indeed, Kiwix is the proper solution for anyone who cares about knowledge access in places where internet connection is spotty. Given the file format it uses (zim) is compressed, and meant to be accessed compressed to pluck a specific article, you can have all the content of English's wikipedia without images and videos in just 36GB. Plus, the specs of the file format are published and it's easy to build your own implementation, I did it for my needs.

It's also great for privacy maximalists : I have wikipedia, wikisource and wiktionary in two languages locally, which means that most of my searches never leave my computer.

Kiwix/openzim don't provide only mediawiki projects either, I've recently downloaded stackoverflow's content (although, I'll need to build a dedicated search engine for it to be really usable).

You can have a look at the incredible amount of available content there: https://wiki.kiwix.org/wiki/Content_in_all_languages

> all the content of English's wikipedia without images and videos in just 36GB

36GB seems like a really big number if it's just text. A cursory Google search says 1MB will hold about 500 pages of text (ignoring compression). So 36GB would be something like 18 million pages? Let's say a 1000 page book is 10cm wide, so 18M pages wind up as 1800 meters of books, or 180 meter-wide bookshelves with 10 shelves each, which is maybe a large library? It seems like a lot of that must be external sources. I wonder what percentage was actually written by Wikipedia editors?

I'd assume that figure would also include the indexes required for searching

Not sure what you mean with external sources, but I have seen nothing but user generated content in there (but I haven't read all wikipedia articles, obviously).

A few things to note, though:

1/ it's not pure text content, it's html content, this has a significant overhead

2/ a zim file is not just compressed content, but also huge indexes referencing where is which content. You look for your article's title in the reference table, you find the position of your article in the file and you decompress just that part. This is what allows for selective decompression without decompressing the whole content.

The zim file format is far from ideal for compression efficiency - all the best algorithms typically don't allow random access without decompressing everything.

Also, wikipedia has a lot of spam and orphan pages, insanely long lists, etc. Those are hard to algorithmically filter out.

Wikipedia (english) currently has about 6.2 million pages https://en.wikipedia.org/wiki/Special:Statistics

I used this in Cuba. It was immensely useful, both to pass the time, and to look up many things of interest along the way without waiting to go to an internet zone.

I was a passenger in a car driving through central Cuba and thought I saw a sign towards Australia. Breaking out Kiwix I found the article and was relieved to see I wasn't going crazy.

Later on I was in an Uber in Australia driven by a Cuban man, and I thought I'd impress my friends with my worldliness by mentioning that there's a town in Cuba called Australia. The driver furrowed his brows and said flatly "no there's not", much to their delight. Can't win em all!

https://en.m.wikipedia.org/wiki/Australia,_Cuba

In 2015 I was curious about the article distribution on Wikipedia so I scraped and visualized some data. At the time I was frustrated by the writing quality and I wanted to know how much of Wikipedia was 'quality' and how many of the 5 million articles were in subjects comparable to a general encyclopedia volume.

My images are here:

https://imgcdn.dev/i/c6fJ8 https://imgcdn.dev/i/c6VRy https://imgcdn.dev/i/c6Wn2

and you can compare that to some Wikipedia hosted comparisons:

https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#/m... https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia

there are many ways to download wikipedia: https://en.wikipedia.org/wiki/Wikipedia:Database_download

It also included great multimedia (images, audio, video, animations). I remember myself spending hours watching all kinds of physics animations.

Reply on news.ycombinator.com