Hacker News Reader

Comment by marginalia_nu | Hacker News Reader

I'll argue any civilized programmer should have a Wikipedia dump downloaded onto their machine. They're surprisingly small, and it saves you from having to use slow and unreliable APIs to do these types of basic processing tasks.

They also let you do less basic processing tasks that would have been too expensive to expose over API.

View on HN

1. Download http://static.wiki/.

2. Run it locally on https://datasette.io/.

3. ???

4. Profit?

I built my own at encyclopedia.marginalia.nu, but basically, yes.

I learned how expensive hashmaps and hashsets are through Wikipedia dumps. I did some analysis of the most linked-to pages. Countries were among the highest. Hash sets for holding outgoing edges in the link graph ended up causing my program to exceed my laptop’s memory. Plain old lists (Python) were fine, though. And given there aren’t a crazy number of links per page using lists is fine performance wise.

Why did lists require less memory? Was it because you only held a subset of keys in the lists?

Lists in Python have a integer for their size and a pointer for each element. Sets presumably have some number of buckets that are used to put pointers in, but many more buckets are allocated than get used in small sets.

Sort them, and use a vector of vectors for the adjacency list... Or better still use a graph processing library or graph database to manage that for you...

This is a fairly large data set indeed. The memory overhead (which is probably something like 4-8x for hash maps?) can start to become fairly noticeable at those sizes.

Since Wikipedia posts already have a canonical numeric ID, if map semantics are important, I'd probably load that mapping into memory and use something like roaringbitmap for compressed storage of relations.

How'd the hashset exceed your laptop memory, if the whole dump is just 22GB? You should be able to fit the entire dataset in RAM.

That is a compressed dump you are looking at. The uncompressed data is much larger. Link graphs in general can grow quite big. Also, not every laptop has 32 GB RAM.

I'm still sticking with 16GB on my laptop so that would exceed my current RAM. That may also cut close for a 32GB machine anyway, since the OS and other programs may not let you access all your physical RAM.

I think my laptop had 8GB at the time

Relatedly: to drastically improve Wikipedia loading speed for personal browsing purposes, do not stay logged in to your Wikipedia account. The reason as explained here (see top reply by baowolff)

https://news.ycombinator.com/item?id=36114477

To be honest, three-tenths of a second per page load just doesn’t matter very much to me. Logging in and out all the time will take much longer.

Well, I specifically stay logged in so that I can opt out of the redesign they dropped a year or so ago. Never made an account before that.

Does the download dump include edit history?

Well there are full database dumps, but they're quite a bit too big to be of much practical use.

I'm usually working with the text-only OpenZim version, which cuts out most of the cruft.

Reply on news.ycombinator.com