Parsing All of Wikipedia to an Offline Encyclopedia | Hacker News Reader

Parsing All of Wikipedia to an Offline Encyclopedia

daveshap.github.io

25 pointskxrm10 comments

This is pretty cool actually. The one thing I always hated about wikipedias .zim files is that they're not really a standard I can use elsewhere. Although I'm having trouble understanding the sorting structure with just one json file I'm looking at. I realize it's sorting by the ID, but is that just simply based off of when the ID in wikipedia was created?

However, I think the json file is extraordinarily nice to have because it's all of wikipedia in just a simple text format. If we could get wikipedia as an entire package with all the html, css, javascript but capable of being run locally, I think that would be an amazing game changer using wikipedia offline, especially on a pi! While kiwix "works" in some ways, it also has some major issues that makes using it such a pain. I'd like for wikipedia to be just a completely source of data and information that I could search and pull up just even a basic html file and read that article from looking it up just in my own directory.

Overall though this is a super cool project!

It's not a hugely useful bit of information, but I found out a while ago that most Kobo eReaders are able to handle ZIM files.

And some of them can have their "on-board" storage increased, which is actually a normally non-user-accessible SD card. I wouldn't do it on a water-resistant model to avoid breaking the seal.

If someone got the entire Wikipedia offline, please do post the size of it.

The author uploaded his plaintext dump here: https://www.kaggle.com/ltcmdrdata/plain-text-wikipedia-20201... it is 22 GB in 605 .json files.

Maybe one could create a full text search for Wikipedia out of this data. This would also allow us to easily gather statistical information from Wikipedia, e.g. how often does some word occur in Wikipedia. It would be great to have a Wikipedia to play and query around with.

This seems like a convoluted method to achieve the end goal, straight up regex parsing pages seems like a last ditch effort not the up front start.

yeah, AFAIK the MediaWiki people spent a lot of time to write a new parser for their visual editor that creates an AST from the syntax. That should be a much better way to create whatever format you want.

OTOH this is just hearsay and I never really looked into it. So maybe this was the last ditch and OP just did not mention the failed attempts before?

tldr; There are 3 parsers for wikitext, 1. the "old" wikitext parser written in php 2. a newer effort written in js and named parsoid 3. parsoid.php which is the one currently in use on (most?) WikiMedia sites.

All the important details are on the wiki page: https://www.mediawiki.org/wiki/Parsoid

Disclosure: I work for the Wikimedia foundation but not directly on the parsing code. Anything I post on hacker news does not represent the views of my employer.

A few years back I remember having had a DVD with all of Wikipedia from a magazine

1st thing on my mind is that handheld Openmoko WikiReader

Reply on news.ycombinator.com