Hacker News Reader

Art institutions take these problems seriously. XFR STN at the New Museum is just one example of dozens of projects which attempt to rescue obsolete digital content. I think the problem is less serious than you think where file formats are concerned. It is very severe with a platform like Twitter, though. An informed public would reject the useless level of threading offered by Twitter...

Do we need to preserve every tweet, though? The important ones get mentioned in articles and other places. 99% is just noise, although I guess it would be interesting for future people to listen in.

It doesn't seem that important of a thing to preserve though. Sure, if we could hear daily conversations of people from the year 1000 we'd learn a lot. But that's only because we don't have access to much other cultural information from back then, unlike what we'll have now with webpages and news articles.

99% may just be "noise", but that doesn't mean it's not worth preserving.

Excluding an entire medium from preservation would be horrible for future research.

I don't see what makes Twitter worth preserving any more than everyday conversation, or packaging labels, or billboards, or whatever other junk information there is. Just because it would be somewhat easier to do preserve a digital thing doesn't make it more important to, does it?

I mean I'm sure it'll be helpful to a subset of researchers. But I don't see why it's Twitter's fault for not making it easy to preserve, as much as it is Microsoft's fault for not preserving WordArt bake sale announcements.

The fact that so many people are wasting relatively large amounts of time producing "noise" is historically relevant.

But we won't need all the noise itself to learn that fact.

Your mission, should you choose to accept it:

Track the evolution of online language as a consequence of Twitter between the years 2001 and 2100.

Which tweets are just noise now?

I'm not sure if you are joking, but I have a linguist friend who mines Twitter for things like evolving language usage. She shares some pretty interesting things that she learns from it, too. I expect that being able to do that with a hundred-year (or multi-decade) corpus could be really interesting.

It must be nice to be able to know what will be important in the future.

I think it ends up being the responsibility of individual families to preserve their digital history as much as their old physical stuff.

For example, I have digitized probably over 5k photographs and several dozen home movies (that I had to record from VHS tapes and even some old film reels), and several thousand pages of documents - report cards, letters, tax forms, etc - of my extend families over the years. That entire archive is now around 20GB big (I have "new" stuff and "old" stuff, pre and post 2000 separate) and I have it backed up in the cloud, on three personal computers, my own external hard drives, and I have it burned in sets of dvds in three places (two homes and one storage unit). I made sure it was all formatted with open standards (extenion-less pdfs, pngs, jpgs, theora, vorbis, odfs, etc). The archives themselves are all UDF formatted.

This kind of stuff is not going to magically become unusable. The formats aren't going anyway. The transport interfaces might, but someone will be able to read dvds, plug usb, and plug sata for several decades at least. And I fully intend to update that collection (probably around 2020) and include the next decade in it in whatever formats are most appropriately modern at the time. At least as long as I still have something like gstreamer to pipe all the audio / video through to transcode it from speex / vorbis to opus or whatever else turns up.

Most families should do something like that. I doubt many of them are.

>Now imagine that those things are stored on floppy disks in Office95 .doc format

I really don't think this is a big problem, in the historical context. Sure, the average person won't be able to casually open those files, but neither can the average native English speaker casually read the original Beowulf!

If a catastrophe wipes out an ancient civilization and all that survives is one laptop-sized cuneiform tablet, historians don't get much from that. If a catastrophe wipes out modern civilization and all that survives is one laptop, historians get gigabytes, maybe terabytes, from that.

Would future civilizations go through the trouble of understanding our video codecs? Machine language?

Isn't there far more complexity in our current stack than beowulf?

Even if data survived, it would take an incredible effort to use.

> Would future civilizations go through the trouble of understanding our video codecs? Machine language?

I would say yes; we go to great lengths to extract information from hard-to-read ancient sources, so future civilizations probably would too: http://www.bbc.com/news/world-europe-31087746

True, but how many people do we have working on that problem?

Compare to the number of people and years of effort that went into building our modern computing platforms. How much research and investment in a very lucrative field? A quick assessment of my university's finances shows they spend twice as much money per diploma on the sciences vs the arts. (Honestly surprised it's that much, though it does exclude medicine which is twice as much as science)

Then consider the hardware and engineers. Just getting to a bit stream is very complex.

I estimate decoding a modern piece of data would be similar to building our entire computerized technology field very nearly from scratch. In my mind that's several orders of magnitude of effort off.

I think there's a middle ground though. For example, Egyptian hieroglyphics were around but not understood until the rosetta stone was found. Without a "digital rosetta stone" those terabytes would be similarly useless.

There's also some historical value in journals of every day folks, e.g. soldier's notes from major battles or those present at historical events.

Given how good people are at cracking things, I seriously doubt it would be that hard to crack a file format.

Well, cracking 100 years from now will probably be much harder. For example, could CSS be as easily cracked if you didn't know it was for DVD video and didn't have access to a working reference implementation?

Either way, we're surely capable of leaving a better legacy than what we currently have.

Even if DVD codecs are 100% opaque to future historians, recall that no video of any format existed until the 20th century, and widespread ubiquitous video recording didn't exist until the 21st. If we're entering a dark age now, what does that make former ages?

Exactly. Note the dearth of photographs of the Titanic. Just a handful, and that was a major project.

I agree that the "dark age" designation is not the best (and perhaps sensational), but the basic issue of how to maintain digital data isn't worth writing off...

Problem: floppy disks and DVDs has a shelf life that's less than 100 years.

How many letters do you have from your great grandparents? For me, it's zero. For records to be preserved, someone has to care about preserving them. For digital data, that means copying it forward to newer media now and then.

I'm sorry, but there's a huge difference in preserving a letter and a file. I say this as someone who has worked in the field of preservation (mostly digital) for close to a decade.

Really simplified... For a letter to be preserved: put it in a box and take it with you when you move and hope that the house doesn't burn down. For a file to be preserved: keep backups to avoid data corruption and media deterioration. Also: control physical media obsolescence and format obsolescence. Most people don't have clue about these things. If our cultural heritage institutions don't get their hands on some of these files, most of them will likely be lost.

And no, it's not only about our great grandparents. It's also about the Turings and Einstens. Their day to day mail correspondence has taught us quite a lot. Turing might not have been a good example though... :)

The problems are simply different, not one is better than with the other. With paper and film, making backups is tedious and expensive and rarely happens. They get lost, mildewed, thrown away, looted, burned, flooded, blown up, eaten by insects, etc. Much of my family history got lost that way. Any piece of paper surviving 200 years is a miracle.

Paper archives have the unfortunate characteristic of concentrating treasures together. 9/11 apparently destroyed quite an archive of photos - we don't even know what all was there. The Vatican Library concentrates a huge collection of paper, and one little event could take it all away.

With digital, copies are cheap. I've been copying forward my old stuff for 40 years now (though I did lose my old IBM punchcard decks and paper tapes!), and it does get easier. With cheap terabyte drives, one drive will hold it all, and I can make copies to make it resistant to catastrophe.

The Vatican Library really needs to make it a priority to scan all those papers.

But this is just a property of acceleration. If languages had started to change at the same speed as file formats have, or we had started to miniaturize in other ways, those millenia old letters might no longer have been usable either. It's definitely worth being conscious of the value of interpreted artifacts, but this is a continuation of a very old tension.