Full-history English Wikipedia dump produced: 5.6TB uncompressed, 32GB 7z'd

Yeah, the deleted articles could probably be automatically filtered, at least on a fail-safe basis by only dumping the ones that have a known "not a legal issue" deletion reason, like "notability" (there are even semi-formalized deletion codes, itself a mild absurdity). There are probably non-technical / non-legal reasons people don't want to dump them, but there's also some of just a "not a priority" aspect. The dump reported in this story is actually the first successful full-history dump in quite some time, because the dump scripts were perennially broken / bogging down due to the size of the data / crashing due to MySQL weirdness. So most of the dump effort has been on just getting the official stuff out. Next up on the priority list will probably be some way of doing image dumps.

You do get a bit of a warning if you recreate a deleted page. When you go to the editing screen at the title of an article that was previously deleted, it'll show you the summary from the deletion log at the top, and ask you if you're sure you want to recreate it. There's also a "nothing can go here" protected placeholder used for articles that are persistently being recreated, which'll make it impossible to edit at that location.

Yeah, I can sympathize on the Wikipedia-inner-sanctum thing. I was actually pretty deeply into it (I've been an admin since '04, was formerly on the Arbitration Committee, formerly active on the mailing lists, etc.), but as the Policy And Process kept accumulating, I lost interest in navigating it, so am more on the periphery these days. It's probably inevitable that things would go that direction, because in the early days there were probably <100 Wikipedians active enough to form the Wikipedia Cabal, all of whom at least recognized each others' names, so stuff could be pretty informal. But it's hard to scale that up to a site with 1700 admins and 15k+ editors. A lot of things are kind of lame about how things are organized these days, but honestly I have no idea how I'd do it better; despite its flaws it's often still amazing to me that Wikipedia works at all.

In the dump, I think for robustness and ease of extracting subsets.

Robustness: Having to essentially play back a log to recover any particular revision increases a chance of something eventually getting corrupted, and so it's somewhat safer to avoid it in something intended to be archival.

Ease of extracting subsets: For researchers, having the revisions be independent allows you to filter the XML dump through a SAX parser (or similar) to grab only revisions meeting particular criteria. If deltas were stored, you'd have to reconstruct those revisions from the deltas, which would make it really expensive to do things like, "I want to look at every article as it appeared at noon on April 1, 2007".

In the live DB, I think just because it's cheaper to get a ton of storage, esp. for rarely-retrieved old revisions, than to add the overhead of computing deltas and applying them to reconstruct revisions. In particular, you'd have to compute a diff for every edit in that situation, whereas currently MediaWiki only computes diffs when a user requests to view one from the "history" tab, which is a tiny proportion of all edits.

Perhaps this is a good time to point to this?

http://thewikireader.com/index.html

Yeah, this is exactly the sort of problem bittorrent was designed to handle

I'm pretty sure LZMA (7z) compresses better than bz2.

At least it does when I test it. But it's slower - sometimes much slower (depends on settings).