The online data that's being deleted
bbc.com
bbc.com
Our mission was to ensure that the assets we were working with were usable by someone ~100 years from now.
We would take something like an LP (Ray Charles was one of my favorites) and play it and record it in a lossless wav at a high bitrate. Then we'd take the physical record, the entire sleeve, and anything else included and scan that at 300dpi in tiff. Of course, we'd document every piece of equipment we used (down to the serial number) with every setting and who did the work. If we later realized a given piece of equipment was misconfigured, we could detect what had been created with it and adjust or re-create as necessary. I wrote some of the standards (XML schemas) for metadata collection which were still in use as late as 2015. (They may still be, I haven't checked.)
Whew.
Then we'd take a version of the audio and images to create mp3s and jpgs for listening+viewing in the reading rooms. The original physical media would go into nitrogen filled vaults in Culpepper, Virginia to prevent oxidization and further deterioration.*
We had similar processes for other media like books, wire spool recordings (popular in WW2), wax cylinder recordings, and pictures around the Civil War. Many of them were exceptionally fragile and required special equipment. My least favorite but most fascinating were Thomas Edison's first motion pictures.
Pop quiz: What does cotton and silver nitrate degrade into? ;)
But in all of those cases, we could go back to the original media which - hopefully - hadn't deteriorated beyond recovery. In a digital-first or digital-only environment, we're counting on fragile components storing rotting file formats on deprecated operating systems in physical locations that may suffer disasters.
* At the time, the RIAA was complaining endlessly and we may have made jokes about RIAA execs and those vaults but I can neither confirm nor deny any such proposals.
Bright light if you look at it wrong?
The best way to archive data long term is to make diverse copies. The easiest way to make diverse copies is to simply make them available to download off of the internet. Don't restrict it to "reading rooms".
Take one of Ray Charles' records..
Is it just the music itself? Does it include the physical media and its condition? If the media has deteriorated, how does that affect it?
Does it include the cover art? Does it include the liner notes? Does it include the notes through the production process?
For the recording of a live performance, does it include the recording equipment and the venue? Does it include the audience reaction?
Does it include documenting everyone who performed backup vocals, accompanied via instruments, produced it, and the label itself?
While normal listeners (99.9% of the public) would stop at the music, the audiophiles will want to understand the physical media and performance conditions, and the researchers/archivists want everything because it's a snapshot of a particular work at a particular time & place via particular means.
But part of your comment is spot on.. diverse sources. It's not feasible for one Library to digitize absolutely everything in the world (or a country), so we participated in an exchange with a bunch (20? not sure anymore) of other libraries, museums, etc who could digitize their own collections and then we could all share them at scale with as little DRM as possible.
Think of it as Napster for libraries. I advocated for no DRM (especially for old, out of production works) but that's why we butted heads with the RIAA.
> the audiophiles will need to understand the physical media and performance conditions,
They really don't :-)
> and the researchers/archivists want everything because it's a snapshot of a particular work at a particular place in time via particular means
Phooey. There are far more interesting things that need archiving.
I once put all my photos through JPEG2 compression, because I thought saving space was a good idea... I've regretted it ever since.
The function of an archive is to preserve artifacts of the past with as much fidelity and context as possible. You can't possibly know what is going to be important to people in the future. Saving 2 cents on storage per century could be the equivalent of using shorter dates/times to save memory, the tradeoff that lead to the Y2K and Y2.038k problems.
Because in theory they already have everything:
"Mandatory deposit (17 U.S.C. section 407) requires the owner of copyright or of the exclusive right of distribution to deposit in the U.S. Copyright Office for the use of the Library of Congress two complete copies of the best edition within 3 months after a work is published. Copies of all works under copyright protection that have been published or distributed in the United States must be deposited with the Copyright Office within 3 months of the date of first publication."
Ref: https://www.copyright.gov/help/faq/mandatory_deposit.html
And final tidbit from me on this.. Yes, they get submissions of all forms of copyrighted material which means they have a pornography collection. When I was there, that group was down the hall and the lead person was a kind old lady that I chatted with every day for months before she revealed her area of focus. It was.. odd.
But the choice remains:
1. archive in as agonizingly irrelevant detail as technically possible
2. archive as much as possible before it decays irretrievably
Can't have both.
They've also mentioned that always knowing the exact technical setup can help track down problems, which reduces the chances of loss as things are archived down the road.
The guy was working in a pretty important national archive, on what sounds like are rare or one-of a kind artifacts. Archives aren't just about preservation, but preservation in a way that allows research to be performed. Research will likely need high quality copies of good provenience, and all the record-keeping could help explain some phenomenon that was introduced during the digitization process.
So, if they just did a quick an dirty rip of as much stuff as they could, they could probably "preserve" more, but that data might be useless to some future research project after the original has degraded into uselessness.
And in any case, my bet is most well-preserved physical objects (at least paper ones) will long outlast any digital copies that are made today.
The medium is the message. (h/t Marshall McLuhan)
The interaction between tools, artists, and expression is endlessly fascinating.
Similarly, other historic audio recordings might contain forensic clues outside the ranges of human hearing that will be useful to future historians.
Finally, storing lossless formats makes sense for an archive, because raw bits are likely to be easier to recover if knowledge is lost, and localized damage to the data will result in only localized damage of the output.
But I also know of the HP archives that burned down while awaiting funding for a pixel perfect proper archiving of it. The result was there's nothing left for future historians.
The Vatican Library is another. It's full of one-of-a-kind documents. They can't afford to do a pixel perfect archive of them, so nothing gets done. It's all at huge risk of fire.
An imperfect copy is infinitely better than ashes.
Is there any published technical document about the choices made by LoC to achieve that goal?
Now that we have kids we back up our photos in multiple places, but we also make sure to print out the ones we really like, just in case. The really good ones we print on canvas and hang on the walls.
But it's true, most of records would be lost if there were a global cataclysm that took out all digital records. That being said, I'm pretty sure our mortgage is only digital so at least we would own our house free and clear!
Very similar to what happens in Mr. Robot where the plan is to encrypt all of the biggest worldwide bank's records so that they're inaccessible, thus wiping out the majority of debt.
If you didn’t watch it yet, were you ever really going to?
Imo, spoilers without warnings are fair after 3 months time+ for TV shows and movies. Maybe even sooner.
I don't know why you'd reach the conclusion that all information falls into the latter category.
Yes. I have the whole thing saved and did in fact plan on watching it eventually.
(This might be a controversial opinion, but if you've seen Fight Club, it's hard to watch Mr. Robot and not know exactly where things are going anyways. Of course, it's definitely a fun watch regardless!)
The only feature it's missing - but I know this would be complicated and advanced - would be to scan for similar images and duplicates that don't exactly match. Ideally it would present them all and then I could manually mark for deletion what I want to get rid of.
So many duplicates. We were all so naive when digital photography first became a thing. I remember back in the day everyone recommended "keep all files, delete nothing, storage is cheap".
"What are my children or any potential grandchildren […] going to do with the 400 pictures of my pet that are on my phone?" Nothing, they don't need it. You sit your grandkid on your lap and look in their eyes and tell them stories about the family dog. They'll be happy to listen, they don't gain much by seeing your 400 photos.
I guess it's a natural human impulse to try to leave a mark for the future. But the future will be okay, they don't need our crap data dumps. There's way more useless fluff being generated than what needs to be preserved. People survived when there was no photography.
Yes, preserve your wedding photos, a few important life events here and there, that can be fun to look at for your kids. The full camera roll on your phone? Who cares? Let's not burden the coming generations with all this stuff. The future is theirs, not ours.
For the few things that you want to preserve, make high quality physical prints. Their huge benefit is that they don't break silently. It's utterly obvious when looking at the physical object that it still "holds your data", as opposed to an HDD, DVD or tape which you need to put in a machine to check if it's still readable, which you won't reliably do. Analog printouts are a very robust preservation format. They can burn, but DVDs and HDDs aren't fireproof either (backup printouts are a solution too). Yes, prints may degrade in color, but it's a transparent process.
But overall, forests need to clean themselves by fires, there needs to be a clean slate. I don't want to know who my great-great-grandparents were. I don't want their family photos. Maybe they were bad people, criminals etc. Maybe there was domestic abuse, trauma, tragedy etc. I don't need to know. I want to live my life with the people that are alive in the now. I don't want my ancestors to haunt me from the grave. And I don't want to haunt my descendants.
It would be fun to know more.
I get the motivation here, but in addition to keeping the stored data, this means one must keep account and authentication systems, maintain a service and UI for people to request this data, etc for years. That seems like an excessive obligation, depending on what the product was which took in the data to begin with. What data should be eligible? If I ordered delivery from a service which folded, should I really be able to request my order records years later?
Even websites can't be assumed to remain at the other end of the bookmark you keep. You MUST download it if you really want an archive you can count on.
If you aren't buying a new mass storage device EVERY year and moving anything older than 3 years to the new storage, you are taking a risk.
This is why I still buy physical books rather than Kindle et al. This is why I convert personal documents and momentos like photos to paper, et al. if it actually matters. I still have photos of my ancestors from the 1900s-1920s only because they are on paper.
And I've worked on most of the advanced technologies in use today over the last few decades - high priest level contact. But the truth you learn if you learn anything is: you should not put technology first in your life! It's a tool but not even the best tool.
One side of my family lost the photos from a leak in the roof. The other side lost the photos because the house was destroyed. (The handful left were ones that were copies given to friends and relatives.)
Paper rots and burns.
Kinda like how conceiving a child is hard if you want one and easy if you don't.
The "war stories for hours" is very strong with this statement, I feel.
I figure that a lot of them can sadly never be told :( but I'd like to firmly +1 the idea of considering what can be shared. Sign me up :P
https://old.reddit.com/r/askscience/comments/5cexmj/how_stab...
Thanks for the link!
I've lost much of my handwritten notes from university.
I used to handwrite code in notebooks before typing it in. That's all gone.
Clearly this isn't yet an example of successfully preserving data. But the mindset of setting a goal to make something readable in the very distant future seems admirable.
We need a cheap storage solution that is reliable. Doesn't need to be fast. Even DVD dont last longer than 10 years.
Someday this whole thing is going to back fire and we move back to hybrid / photobooks.
> Microsoft and Warner Bros. have collaborated to successfully store and retrieve the entire 1978 iconic “Superman” movie on a piece of glass roughly the size of a drink coaster, 75 by 75 by 2 millimeters thick.
> The hard silica glass can withstand being boiled in hot water, baked in an oven, microwaved, flooded, scoured, demagnetized and other environmental threats that can destroy priceless historic archives or cultural treasures if things go wrong.
> “One big thing we wanted to eliminate is this expensive cycle of moving and rewriting data to the next generation. We really want something you can put on the shelf for 50 or 100 or 1,000 years and forget about until you need it,” Rowstron said.
https://news.microsoft.com/innovation-stories/ignite-project...
The big question I suppose is whether you can consumerize it. Though I suppose you could imagine the return of something like mall photobooths: you buy an organic dye substrate which something like a CD drive can write to, then drop it off to be "developed" into an impervious substrate by a harsher process (guessing there's an NH4F etch in that process).