Hacker News Reader

Comment by giancarlostoro | Hacker News Reader

To be fair wikipedia compressed without images is pretty small. A couple tens of gigs no more than 80. You can download it to your phone and there are apps to view it all. Not sure how larger it is decompressed but 2 TB sounds like overkill. But yeah wikipedia would be a good way to test it.

Also did you mean every single wikipedia site or just the English one? I'm thinking just the English one's sufficient.

View on HN

I know what you mean with "sufficient",but English pages are often lacking informations on a lot of subjects related to countries outside the anglosphere.

I still don't understand why wikipedia articles aren't translated according to a common script, at least where there are featured translations (marked a star)

Feel free to make that happen.

Also, did he mean the entire revision history of Wikipedia, or just the most recent versions of pages?

That would definitely explain 2TB haha but yeah, if you pull the recent English version without images it's like ... I wanna say 30GB off the top of my head.

As for anyone curious:

https://dumps.wikimedia.org/backup-index.html

The most recent was 13.9GB compressed and ~65GB uncompressed. The version without images, talk pages, and revision history is 'enwiki-20180320-pages-articles.xml.bz2' here:

https://dumps.wikimedia.org/enwiki/20180320/

The article text contains Wikipedia markup which is a bit difficult to remove but not impossible, there are some existing projects for doing that. DBPedia would have the raw text, but it's not nearly as current.

You can get the raw text with wikitext and whatnot removed from the search dumps made every week (also has various other metadata)

http://dumps.wikimedia.your.org/other/cirrussearch/

These are structured with a JSON string for each doc roughly like https://en.m.wikipedia.org/wiki/Foobar?action=cirrusdump

Reply on news.ycombinator.com