Download the Entire Wikimedia Database
dumps.wikimedia.org
dumps.wikimedia.org
Personally, I think Wikipedia's quality is too poor for that. Plus, it's digital, so when our civilization is at risk of "[losing] the entire library" it probably would have already lost the ability to maintain the computer systems to access Wikipedia dumps.
But as long as it continues to exist, some future civilization could figure out how to read the data again, eventually. Just like we eventually discovered how to read ancient languages that were once forgotten.
Eh, I think you're vastly underestimating how difficult that would be.
1. The media would have to last hundreds of years at least, when it's hoped modern archival media can last maybe fifty.
2. Even assuming the media did last, the new civilization would have to reverse engineer encoding on top of encoding on top of encoding (e.g. physical disk encoding, complex filesystems, file formats, character encodings). Our civilization already has trouble reading some old file formats, and our disks already have trouble reading their data (which is why they pack a ton of error correction information).
It took the Rosetta stone to figure out how to read encoding of Egyptian hieroglyphics, when that language was still alive in the form of Coptic.
3. Then you're dealing with the probability that the hard disks the future archeologists find will even have a Wikipedia dump on them. That probability will be very small, given very few people will download these dumps.
As far as media goes. That's true, but its a bit of a numbers game. After all you only need one unusually preserved specimen. The dead sea scrolls survived after all. Not to mention intentional preservation efforts. I know github has its artic vault thing. There's even a copy of wikipedia on the moon! https://meta.m.wikimedia.org/wiki/Wikipedia_to_the_Moon/Wrap...
IIRC, Coptic is directly descended from Ancient Egyptian, but the Rosetta Stone was still needed to decipher hieroglyphics.
It won't be as simple as you think. The problem will be more like: here's 10TB of partially corrupted binary data, find the text when you don't know the encoding (oh, and the text may be compressed with an algorithm you also don't know).
You should see some of the crap in the books at Alexandria: The world is flat and there are four elements and other bollocks. Obviously I'm taking the piss. The content is sometimes just as important as the factual accuracy of the content. For a given value of factual accuracy.
WP is written by people and holds a vast amount of stuff. It is flawed in my opinion in many ways but that is the human experience.
I live in a town called Yeovil in Somerset, UK. https://en.wikipedia.org/wiki/Yeovil . Several years ago I noticed an incorrect old name for the place and I tried to correct it. I appealed to the Domesday Book which is considered quite authoritative hereabouts but I linked to the only site I could find which sold copies of it. My edit was thrown out by the local Somerset editor rather than being fixed. I own a coffee cup bought from the local museum that lists the >60 spellings of this tiny town over the last 1500 odd years. The editor wouldn't accept that either "original research" WTF! I didn't put the names on the mug - archaeologists, historians and a bunch of tax gatherers hired by a hoard of Normans back in the day did that. OK, no they didn't - they scrawled stuff and the museum gathered together the scrawls and made my mug. I did one for a tourist shop on the Plymouth Barbican wrt the Mayflower complement, about 30 years ago. To be fair, I simply copied the names off the board near the Mayflower steps!
My point is that WP is what it is and you need to see it for that. It is both a store of knowledge and also a store of knowledge and blatant lies and everything in between ... about knowledge. It contains its own metadata and also omits vast amounts of it.
WP is without question in my mind absolutely magnificent but you do need to learn how and when to interpret it to fit in with your idea of factual - whatever that is.
But that reflected the actual state of knowledge at the time, which is what you'd really want to study.
Let me put it this way: Wikipedia wouldn't even allow Wikipedia to be used as a source for one if it's articles, because it's too unreliable: https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources#Use...
Er.. I think you're missing the point of citing sources. If an otherwise unverified claim could cite as its source another unverified claim, it would make citing sources meaningless. If a wikipedia article wanted to cite a verified claim from another wikipedia article that did have a verified source, they may as well just use that original source as their source too.
I think your example actually shows wikipedia in a very good light.
1) The Wikipedia guideline doesn't mention that logic, 2) quite a lot of Wikipedia doesn't cite anything, and 3) it not unusual for a passage in Wikipedia to not actually be supported by the citation given (e.g. https://en.wikipedia.org/w/index.php?title=Special:WhatLinks...).
In any case, this is getting off topic. My point was that, if you're looking for an ark of cultural knowledge to survive some apocalypse, Wikipedia's a bad choice. IMHO, something like Project Gutenberg plus a newspaper archive would be about 1000% better, and take up far less space. If you got space to spare, throw in Libgen. Wikipedia's not a replacement for its sources, and I pity the future scholar that would have to rely on it without being able to check them.
But unless your data storage is orders of magnitude more reliable than any current technology and you package your archive with an equally reliable computer to read it, the concept of a digital ark fails. If you don't to that, your archive will be unreadable, and and unreadable archive is useless.
https://wiki.kiwix.org/wiki/Content
Their Wikipedia bundle wasn't being updated for a while and had fallen out of date, but that seems to have been fixed now.
That would be a great post-apocalyptic knowledge archive, and it would be easy enough to recharge using a USB solar panel.
This is one of the nicest hidden features/easter eggs I've ever come across. Thanks for sharing this!
How's the search functionality in wikipedia when used like this?
Example: https://www.mobileread.com/forums/showthread.php?t=276219
The amount of ancient literature that is no longer available is vast. History, literature, you name it, so much has been lost. The odds are very good that at least a significant portion of that were in the Alexandrian library.
Curious if you ever compared how PageRanks correlate to traffic? (They make their per-page traffic available too.)
It would be interesting to see the largest disparities -- super-popular pages in visits but which don't have nearly as many internal Wikipedia links to them, versus unpopular pages but that have tons of internal Wikipedia links to them.
https://www.nayuki.io/res/computing-wikipedias-internal-page...
Geographic coordinate system is first.
But it's still a fantastic and incredible piece of software. When it gets to the point where I can portably keep the full 60gb zim file seemlessly, it will change simple computer for low broadband areas. Imagine the uses as a portable and versatile database that could accept json, html/css, data to make your own offline encyclopedias!
Not according to Wikipedia. Wikipedia much rather beg people from all corners of the world to donate, than restricting access to their data. That's what a good, honest and well-meaning foundation does.
And yes, no sane person shuffling a lot of data around is using AWS because of their awful bandwidth pricing, Wikipedia included.
I suspect the traffic caused by this is minuscule compared to the overall traffic, anyway. But that's just a guess.
Almost certainly not. I doubt this is even a rounding error on their budget.
I'm not sure how this stuff works at a high level (IANA network engineer), but i think they have peering presence at internet exchange points - https://www.peeringdb.com/asn/14907 so im not sure they actually pay for bandwidth at all, at least on some routes
> Backup dumps of wikis which no longer exist [...] This includes, in particular, the Sept. 11 wiki.
There was a Sept. 11 wiki hosted by Wikimedia?
When I follow the links I find 2012-2013 data but may be I missed something?
https://meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_...
I suspect if you had some genuinely good reason why you wanted it, and asked real nicely, and provided some way to transport the data, you might be able to make an arrangement of some sort to get it.
In a way all articles should belong to a category...
There are collections of article, but for example there are no collections of all articles related to computer science or some part of Swedish history.
I think labelling is useful when you can download individual articles (by label) efficiently. I don’t think it’s practical to create torrents for each nor efficient to ask users to scrape. IPFS could have helped if it wasn’t so slow in practise.
Also see https://wiki.kiwix.org/wiki/Content_in_all_languages
Dewey categories are a good way to label things, I guess.
My reason for saying this is one of the only things you can do to ensure lasting democracy is that it is in the realm of what is possible in a physical sense that at some point through some mechanism the online version simply does not inform the public on some important public issue, whereas the history as you can download it today does. Though, I wouldn't speculate about what the mechanism might be or what kinds of subject.
At that point in a physical sense you could consult your offline copy on an airgapped PC or future equivalent and I think it would be impossible for any group of any kind to even know you were doing that let alone stop it.
How you might get the word out is another question but having this personal capability is easy for the people here, as technical users and simple consumers. Indeed the whole entire Internet was set up as a distributed network in case of nuclear attack, so the entire topology of the Internet is set up for you to do this easily today.
It's a click and a cheap flash drive or slightly more expensive microsd card away. You can take this step in less than 20 active minutes of your time and for less than $50 if you go with an external spinning disk drive (such as 1 terabyte) or $200 or so if you go with a microsd card. It doesn't really matter if the file ultimately fails, this is not a critical backup for you to have just a nice to have. You could write the file's checksum onto the drive in marker so you can tell whether it's still correct later (as opposed to having bit errors).
Maybe there is some file type that has a bit of redundancy (checksums) for long-term storage, since due to the large amount (several hundreds gigabytes) I wouldn't be all that surprised if a few bits flipped over the course of several years in cold storage. But I don't know what kind of file type has any sort of redundancy or parity built into it that is supposed to protect against this. (Does anyone know?) Most likely the hash just wouldn't match what you wrote in pen on it but it would still be useable.
Regarding choice of spinning disk or microsd card: I guess it's in the realm of what's possible in a physical sense that at some point people would have their personal property rummaged through by some group and a hard drive is pretty obvious and could be stolen or removed for that reason. (In a physical sense, not speculating about social or political developments that might lead to that.)
So for this reason perhaps best would be to put it on a microsd card even though it is quite a bit more expensive. I guess written once, bit rot causes microsd cards to decay within a few years if not used at all.[1] I don't know for spinning media but I guess it's also about 5-10 years at least.[2]
You could put the microsd card under a postage stamp for example and put an important unrelated document into the envelope, which you would expect to keep for many years. Of course you could always end up accidentally discarding your envelope (while retaining its contents) but that risk shouldn't matter too much. In a physical sense it is possible for groups to x-ray all paperwork (such as envelopes as I just suggested) and a microsd card's electrical contacts are pretty obvious in an x-ray. (It looks like this [3]). I don't have any suggestion that works against this attack, which is within the realm of what's possible according to the laws of physics.
I'm not speculating on what social or political developments might possibly make anything like this necessary at some point in the future, but we still live in a world governed by the laws of physics so as technical professionals you have a huge leg up on most of the world. Spending $50 doing this today might save democracy tomorrow. You could also leave it as a time capsule however the storage longevity is not that long (between 5 and 20 years I guess), and in a physical sense, a time capsule is not particularly secure and would require instructions for someone else to figure out so it's not great in that sense.
So in terms of what you can do today, I would suggest just getting an external 1 terabyte usb drive ($50), downloading the dump together with history (20 active minutes), writing the checksum onto it in marker and just putting it somewhere. Obviously this small $50 investment is one you would hope never to have to use, but who knows, you might go down in history as the one who saved some small part of the world. Though, obviously, not in Wikipedia history.
[1] https://www.quora.com/What-is-the-longevity-of-a-sd-memory-c...
[2] https://serverfault.com/questions/986911/how-long-will-unuse...
[3] https://www.reddit.com/r/pics/comments/3b6bjw/i_xrayed_an_sd...