Comment by caseysoftware | Hacker News Reader

Comment by caseysoftware | Hacker News Reader

I started at the Library of Congress 20 years ago this month working on their digitization effort. It was HUGE even then - an estimated 50TB/day needed to be digitized - and that did not address the 200+ years of backlog or the advent of blogging (rare at the time), streaming media, and everything else online.

Our mission was to ensure that the assets we were working with were usable by someone ~100 years from now.

We would take something like an LP (Ray Charles was one of my favorites) and play it and record it in a lossless wav at a high bitrate. Then we'd take the physical record, the entire sleeve, and anything else included and scan that at 300dpi in tiff. Of course, we'd document every piece of equipment we used (down to the serial number) with every setting and who did the work. If we later realized a given piece of equipment was misconfigured, we could detect what had been created with it and adjust or re-create as necessary. I wrote some of the standards (XML schemas) for metadata collection which were still in use as late as 2015. (They may still be, I haven't checked.)

Whew.

Then we'd take a version of the audio and images to create mp3s and jpgs for listening+viewing in the reading rooms. The original physical media would go into nitrogen filled vaults in Culpepper, Virginia to prevent oxidization and further deterioration.*

We had similar processes for other media like books, wire spool recordings (popular in WW2), wax cylinder recordings, and pictures around the Civil War. Many of them were exceptionally fragile and required special equipment. My least favorite but most fascinating were Thomas Edison's first motion pictures.

Pop quiz: What does cotton and silver nitrate degrade into? ;)

But in all of those cases, we could go back to the original media which - hopefully - hadn't deteriorated beyond recovery. In a digital-first or digital-only environment, we're counting on fragile components storing rotting file formats on deprecated operating systems in physical locations that may suffer disasters.

* At the time, the RIAA was complaining endlessly and we may have made jokes about RIAA execs and those vaults but I can neither confirm nor deny any such proposals.

This excessive detail seems nutty to me, and ultimately counterproductive. It's counterproductive in that it reduces the amount of things that can be preserved. What's more important - audio detail that cannot be heard, or more records?

The best way to archive data long term is to make diverse copies. The easiest way to make diverse copies is to simply make them available to download off of the internet. Don't restrict it to "reading rooms".

I think you want both - high fidelity and many copies and diverse storage locations.

Me, no, I can't think of a use for the super hi fi version. Heck, just the snap crackle pop on the best vinyl pressings destroys any utility of super hi fi.

It comes down to "what is the work?"

Take one of Ray Charles' records..

Is it just the music itself? Does it include the physical media and its condition? If the media has deteriorated, how does that affect it?

Does it include the cover art? Does it include the liner notes? Does it include the notes through the production process?

For the recording of a live performance, does it include the recording equipment and the venue? Does it include the audience reaction?

Does it include documenting everyone who performed backup vocals, accompanied via instruments, produced it, and the label itself?

While normal listeners (99.9% of the public) would stop at the music, the audiophiles will want to understand the physical media and performance conditions, and the researchers/archivists want everything because it's a snapshot of a particular work at a particular time & place via particular means.

But part of your comment is spot on.. diverse sources. It's not feasible for one Library to digitize absolutely everything in the world (or a country), so we participated in an exchange with a bunch (20? not sure anymore) of other libraries, museums, etc who could digitize their own collections and then we could all share them at scale with as little DRM as possible.

Think of it as Napster for libraries. I advocated for no DRM (especially for old, out of production works) but that's why we butted heads with the RIAA.

Congress can help by allowing no DRM for archival use.

> the audiophiles will need to understand the physical media and performance conditions,

They really don't :-)

> and the researchers/archivists want everything because it's a snapshot of a particular work at a particular place in time via particular means

Phooey. There are far more interesting things that need archiving.

I strongly suspect that storage of data is far cheaper than the costs of digitization, or actually storing physical objects, regardless of how high the resolution.

I once put all my photos through JPEG2 compression, because I thought saving space was a good idea... I've regretted it ever since.

The function of an archive is to preserve artifacts of the past with as much fidelity and context as possible. You can't possibly know what is going to be important to people in the future. Saving 2 cents on storage per century could be the equivalent of using shorter dates/times to save memory, the tradeoff that lead to the Y2K and Y2.038k problems.

It's not remotely possible to store everything. Choices have to be made. Archivist time is expensive, too.

The Library of Congress already has it decided what they need to store, at a minimum... "The Library of Congress"

Yes, that was the goal but they were prioritizing works that may be lost due to physical deterioration.

Because in theory they already have everything:

"Mandatory deposit (17 U.S.C. section 407) requires the owner of copyright or of the exclusive right of distribution to deposit in the U.S. Copyright Office for the use of the Library of Congress two complete copies of the best edition within 3 months after a work is published. Copies of all works under copyright protection that have been published or distributed in the United States must be deposited with the Copyright Office within 3 months of the date of first publication."

Ref: https://www.copyright.gov/help/faq/mandatory_deposit.html

And final tidbit from me on this.. Yes, they get submissions of all forms of copyrighted material which means they have a pornography collection. When I was there, that group was down the hall and the lead person was a kind old lady that I chatted with every day for months before she revealed her area of focus. It was.. odd.

Yes, I get that The Library of Congress Hath Spaketh.

But the choice remains:

1. archive in as agonizingly irrelevant detail as technically possible

2. archive as much as possible before it decays irretrievably

Can't have both.

Odds are the technical information about the recording setup is fairly constant, and thus doesn't require much manual input, nor storage space to consume.

They've also mentioned that always knowing the exact technical setup can help track down problems, which reduces the chances of loss as things are archived down the road.

Been saying this about Wayback Machine for awhile. They've archived every HTML page ever, but it's weighed down by high-res images, javascript, CSS, and more recently video. All of this is a liability to preserving the HTML itself. I'm not saying that graphics and video aren't valuable, but it would be nice if they took a more defensive posture to the collapse of the internet. If they are in danger of shutting down entirely someday, will anyone be able to "at least" grab all of the PDFs and HTML and TXT docs that IA has archived? Or will we have to copy tons and tons of extraneous stuff too?

> This excessive detail seems nutty to me, and ultimately counterproductive. It's counterproductive in that it reduces the amount of things that can be preserved. What's more important - audio detail that cannot be heard, or more records?

The guy was working in a pretty important national archive, on what sounds like are rare or one-of a kind artifacts. Archives aren't just about preservation, but preservation in a way that allows research to be performed. Research will likely need high quality copies of good provenience, and all the record-keeping could help explain some phenomenon that was introduced during the digitization process.

So, if they just did a quick an dirty rip of as much stuff as they could, they could probably "preserve" more, but that data might be useless to some future research project after the original has degraded into uselessness.

And in any case, my bet is most well-preserved physical objects (at least paper ones) will long outlast any digital copies that are made today.

The excessive detail is a necessity and a precaution. Researchers need that information (textual genetics or any art history for instance, maybe future archeology). A message can't be separated from the form it embodied; you need both to understand how an artwork has been received. Besides, you always want to preserve more information than what is needed today, just in case new ways to exploit data appear later. That's what archeology does when it refuses to exploit some sites and preserves them for later : maybe the current techniques destroy data, so we save some sites for later - just in case.

> A message can't be separated from the form it embodied

The medium is the message. (h/t Marshall McLuhan)

The interaction between tools, artists, and expression is endlessly fascinating.

There was a Woody Guthrie recording (that I have not heard myself) on wire spools that was released in the 2000s. If I recall correctly, the audio restoration process was only possible by recording the audio with a 384kHz sampling rate, because the wire spools had picked up ambient EMI that could be used to correct for speed variations. If the only available samples were MP3s, then the audio restoration could not have been done.

Similarly, other historic audio recordings might contain forensic clues outside the ranges of human hearing that will be useful to future historians.

Finally, storing lossless formats makes sense for an archive, because raw bits are likely to be easier to recover if knowledge is lost, and localized damage to the data will result in only localized damage of the output.

I know the professional archivists strongly disagree with me on this. I know there are cases like the one you mention where higher precision turned out to be useful.

But I also know of the HP archives that burned down while awaiting funding for a pixel perfect proper archiving of it. The result was there's nothing left for future historians.

The Vatican Library is another. It's full of one-of-a-kind documents. They can't afford to do a pixel perfect archive of them, so nothing gets done. It's all at huge risk of fire.

An imperfect copy is infinitely better than ashes.

> Pop quiz: What does cotton and silver nitrate degrade into? ;)

Bright light if you look at it wrong?

> Our mission was to ensure that the assets we were working with were usable by someone ~100 years from now.

Is there any published technical document about the choices made by LoC to achieve that goal?

Reply on news.ycombinator.com