Tell HN: Kiwix and ZIM – some Big Data for the little guy

I found myself needing a great deal of information about the English language for a learning project I’m working on. I was using python get the information from Wiktionary’s site, process it, and use it for creating lists of English words. I wanted access to Wiktionary’s vast pool of information but realized that it would be impolite for me to slam their servers over and over with requests, and what if I were blocked? I looked in to several solutions including grabbing a json dump of the site and trawling through it.

I discovered Kiwix for Linux and the ZIM file format. Kiwix when combined with the ZIM file of the English Wiktionary allows me local web access to essentially the full wiktionary site. While the ZIM is on an SSD it allows fast unlimited access to the site it mirrors. Kiwix ZIMS are available for lots of websites including Wikipedia and project Gutenburg:

https://wiki.kiwix.org/wiki/Main_Page