Hacker News Reader

Also, did he mean the entire revision history of Wikipedia, or just the most recent versions of pages?

That would definitely explain 2TB haha but yeah, if you pull the recent English version without images it's like ... I wanna say 30GB off the top of my head.

As for anyone curious:

https://dumps.wikimedia.org/backup-index.html

The most recent was 13.9GB compressed and ~65GB uncompressed. The version without images, talk pages, and revision history is 'enwiki-20180320-pages-articles.xml.bz2' here:

https://dumps.wikimedia.org/enwiki/20180320/

The article text contains Wikipedia markup which is a bit difficult to remove but not impossible, there are some existing projects for doing that. DBPedia would have the raw text, but it's not nearly as current.

You can get the raw text with wikitext and whatnot removed from the search dumps made every week (also has various other metadata)

http://dumps.wikimedia.your.org/other/cirrussearch/

These are structured with a JSON string for each doc roughly like https://en.m.wikipedia.org/wiki/Foobar?action=cirrusdump

Reply on news.ycombinator.com