Hacker News Reader

But digital copies ain't always perfect - you forget about bit rot [0] unfortunatelly. One bit flip on stream of compressed picture or archive distorts remaining result. Entire digital world is founded on analog physical world with all its imperfections. Of course there are solutions to mitigate them: checksuming, redundancy, error correction. But how often one checks checksum hash on copy? Edit: CRC doesn't count as it is too weak.

I agree on open standards and deduplication, I hope content addresable archiving will get more mainstream, IPFS brings big hopes.

[0] https://en.m.wikipedia.org/wiki/Data_degradation

> But how often one checks checksum hash on copy?

Your operating system does this.

Only if you use ECC RAM and ZFS/Btrfs/ReFS. CRC e.g. in TCP is too weak.

Ah this is right, I was thinking of removable media, cdrom/dvd usually do this.

Yes, if one was going to reliably archive information they wouldn't use FAT32 on a single spinning disk. Does this really need to be pointed out? The point is digital gives you ways to have perfect copies, correct loss, and the ability to transfer to the next storage medium with no decay. Things analog could never achieve.

Hybrid analog could achieve this. Film could have a picture printed on it, but instead of going for maximum resolution (down to grain level) it could have tiny tiny pixels, not visible to the eye, but readable by a computer. A kind of steganography, if you will.

The data encoded in the steganography, could be any data of course, but for our purposes, it would be the image itself.

Such a picture could be copied, with loss, in an analog fashion, but could also be copied perfectly in a digital fashion, by a digital reader capable of reading out the steganography information. Even the analog copy could be digitally read if it was of high enough quality.

Another, more straightforward, way would be to print on the film, first an analog rendition of the picture, then a digitally encoded pattern of the same image, rinse repeat.

But from a future safe perspective, it could be smart to encode the data itself inside the image, as per the first paragraph. Someone copying the images could unwittingly copy the digital data too!

You might be interested in Optar http://ronja.twibright.com/optar/

Indeed, thanks.

> The data encoded in the steganography, could be any data of course, but for our purposes, it would be the image itself.

This was an intriguing idea, but after I did some back of the envelope rough numbers, I tentatively concluded microform might not encode densely enough to accomplish that. It already has poor fidelity characteristics for photographic data. Right off the bat you are losing lots of information during the reducing process, and the film itself does not have the kind of qualities photographers look for. Color microform is also much more expensive than black and white. So the color film works well for newsprint photos, but for archival grade purposes, we're likely better off with the original film negatives. For deep storage archives, I could see a use case for microform, and with the encoding you suggested, but not quite at one picture per frame (page).

Steganography relies upon imperceptibly embedding information into the picture, so there is a very low encoding density limit because we're working with analog pictures. Optar at 600 ppi uses an entire page to encode 200 Kb, but microfilm is closer to 300 ppi, so at 100 percent density with no picture, we can expect around 100 Kb per page of Optar-encoded binary data. An average smartphone picture size these days is around 5 MB, closer to 15-50 MB for DSLR pictures.

Optar-encoding the smartphone picture, we are around 50 pages before we scatter it into the picture to steganographically encode it. Already we can see the picture has to be blown up across more than 50 pages and stitched back together. At imperceptible levels of encoding density, I'm guessing we're looking at 500+ pages, maybe. A standard microfiche holds about 98 pages, so we're talking about 6'ish fiches per picture at cellphone grade, much more for DSLR grade picture encoding.

For high quality archival photograph storage, I think current librarians have it right: lots of duplicates of negatives, geographically separated, then service copy negatives off those masters, digitally scan in the most-used pictures off the service copy, and let patrons use the digital copies. Migrate the digital copies until we establish a universally-accepted digital bootstrap mechanism and process for Long Now-span time scales.

> bit rot

Isn't it specific to magnetic media only? (I don't think optical discs susceptible to bit rot. Especially M-discs which are said to last for a thousand years.)

The general concept isn't specific to magnetic media at all, although the underlying mechanisms vary.

Any EPROM/EEPROM/Flash will leak charge from its cells and eventually lose data if not periodically refreshed. Even some mask ROMs develop bad bits over time due to latent manufacturing defects.

Many CD-Rs and DVD-Rs recorded ~15-20 years ago already have uncorrectable errors (due to the dye degrading?). Pressed discs are nominally immune, but may have manufacturing defects that lead to progressive failure (e.g. "laser rot" on LaserDiscs manufactured by certain plants).

I was thinking about this the other day.

Would a good solution for /really/ long term storage be a kind of WORM ROM based on fuses that could be burned to 'program' a given area? (I think some EPROM might be this, but I'm talking about more storage density than even current top end flash drives.) Obviously you'd only want to use this for the LONG term storage, but if it's cheep enough I could see it being useful in the same use cases that existing optical media were and even in roles like backup / audit logs.

This is a complex problem.

https://www.archives.gov/preservation/technical/storage-stra...

https://en.wikipedia.org/wiki/Digital_preservation

The Internet Archives preferred bmedium for book storage is ... books.

https://www.theguardian.com/books/2011/aug/01/internet-archi...

Quartz glass is a proposed medium, with a potential life of 300 million years.

https://www.scientificamerican.com/article/data-saved-quartz...

One major benefit of microfilm in my book is precisely the simple tech - even after the zombie apocalypse, just about anyone will be able to quickly figure out how to read what's on them. (Yes, 'zombie apocalypse' is just a placeholder for any massively disruptive event...)

(That being said, digital copies would be way more useful here and now; let's do both!)

There was an effort in the 1950s and 1960s to stock major fallout shelters with a set of microfiche on how to make just about everything, along with readers that just needed sunlight. Wonder if any of those are still around.

(This wasn't a joke back then. Everybody remembered the wrecked cities of WWII. They had to be built back up. You might need those instructions.)

> Everybody remembered the wrecked cities of WWII

Is this actually a thing in American memory, given that no American city was bombed in WWII?

I think it's fair to say that American soldiers brought back those memories. My grandfather served in the Army towards the end of the conflict in Europe and, when he would speak about it, he spoke about seeing destruction. (It was not something he spoke about easily or lightly.)

>> Everybody remembered the wrecked cities of WWII

> Is this actually a thing in American memory, given that no American city was bombed in WWII?

To veer off-topic a bit, I think this is an interesting question brings up two important points:

First, as the WWII generation has died out, people seem to have become less averse to war and to care less about the institutions built to prevent it (the UN, EU, international law, etc.). They were built because the survivors of WWII thought war was the worst scourge of humanity, and that another war - with ~1945 technology - could destroy civilization.

Second, the U.S. experienced very little direct harm in WWII, and even the overseas casualties were orders of magnitude less than those of the European and Asian countries (e.g., the U.S. lost 200,000; the USSR lost 20 million). If the worst happens and the U.S. fights another major war, the enemy likely will be able to strike the U.S. with missiles. Even North Korea can do it to a limited degree.

With the loss of the WWII generation, I think Americans have become cavalier about war, and because the WWII didn't affect the U.S. nearly as badly as the Europe and East Asia, they don't conceive of a different kind of war, such as one where missiles rain down on every American city. We can assume the Bay Area would be a priority target, for example; it's hard to imagine what that would be like; it's hard to imagine the consequences of it.

Ideally let's also have various colleges create new versions of the source instructions and collectively create a better modern set to use as bootstrapping (and feed this back as the foundation for future education as well).

> the fact that data can be copied perfectly, as many times as you want

Tangentially, this makes me wonder: While microfilm can't be copied perfectly, isn't it true that the source quality (e.g. the crispness of the print) is vastly lower than the resolution of the medium itself? Which seems like it would imply that the "lossy" transformation might in some sense still lossless, at least for a while - like converting a high bitrate mp3 recording of a radio broadcast into a lower bitrate one. Or perhaps the kinds of loss are different. I'm sure I'm confusing a lot of ideas and butchering some terms here.

I think you're trying to say something, but I'm getting lost in the details. Can you elaborate more on what you mean by source quality / crispness of the print / resolution of the medium itself? What items are you comparing? Please be specific.

With arbitrary numbers: if the original paper was printed at 100dpi, and the film captured 1000dpi, then the losses from copying that film might not make a real difference in fidelity.

Sure, so long as you assume the transfer is 100% perfect with no errors or artifacts and can make subsequent copies similarly. In practice, few physical processes can offer that level of assurance, and digital copies can.

In the analog world, you have to think about signal-to-noise ratios instead of DPI. Any given copying technique will add some small but nonzero amount of randomness to the output, which destroys some information.

The way digital copying gets around this problem is by having predefined signal levels that are farther apart than the vast majority of copying error, and writing a new master every time instead of the analog copy.

The first copy is unlikely to, since a random error (i.e. an unintentionally developed silver grain) will only introduce a distortion 1/10th the size of the original smallest detail. But what about after 10,000 copies? Scattering random 1/10th sized blobs all over the original details is still going to do something to the experience.

You're right! Copying to a higher-definition medium should in theory allow for effectively lossless copies.

There is a potential catch, in that analog copies will tend to introduce some amount of physical errors no matter how much the hypothetical increase in definition should allow for perfection. Subsequent copies, no matter how high their quality, will both tend to amplify these errors and introduce new ones. You get a similar result if you repeatedly re-endcode data even in high-fidelity lossy ways - each encode introduces artifacts and amplifies previous artifacts.

It's an intriguing idea! Thank you for bringing it forth for discussion.

Microfilm is pretty high resolution. How much does a 10K scanner go for these days?

One big advantage of film (especially transparencies rather than negatives) though is that it can trivially be read, regardless of technology or hardware.

I can view on a light table or project a photographic slide from 1935 just as easily as I can view a slide developed yesterday. On the other hand, it's not a trivial task to read a magnetic tape from the 70's. These days it's not even trivial for me to read a CD, I don't own a CD reader, and if I'm away from my desk I can't even read a USB flash drive on my Macbook Pro.

That's important for casual readers, but less important for historical reconstruction purpose. Consider that reseachers used a micro-CT scan followed by volume cartography analysis to locate the ink within the thoroughly burnt-to-a-crisp En-Gedi scroll [1]. Compared to that, a CD is nothing.

https://en.wikipedia.org/wiki/En-Gedi_Scroll

If you don't know the encoding the CD uses, getting the bits out is the easy part.

You are correct, that digital documents may be copied perfectly. However, I don't think the author "failed to emphasize" that topic because that wasn't the goal of the essay.

It is part of the "Object Lesson" series - "Each Object Lessons project will start from a specific inspiration: an anthropological query, ecological matter, archeological discovery, historical event, literary passage, personal narrative, philosophical speculation, technological innovation—and from there develop original insights around and novel lessons about the object in question." - http://objectsobjectsobjects.com/about/

The object in question is microfilm readers, and the things we might learn from it. Not digital technology, nor the reasons why microfilm went by the wayside.

> ...but never forget that what we gain is the tremendous ability to copy perfectly.

Yeah, all except the first one. I'm not gonna argue with the gushing passion for all things digital in the rest of your comment, but I will first of all, point out that putting your photos, music, and videos in the cloud is absolutely not an exact copy--cloud providers don't have ToS that cover the compression that they apply to your media. There's no telling how badly they'll compress it to make room for the newest stuff years from now.

Since you offered such an untempered view of digital media as the ultimate achievement of human civilization, I'd like to offer an alternative viewpoint that recognizes that digital copies reduce very deep, meaningful experiences down to very meaningless bits. That's great for newspaper articles, but it's no way to live life. Just yesterday I was sitting in the park under a tree. The winds were blowing, thousands of leaves dancing above, and distantly a church bell began tolling, it's bang-clang rolling and washing through and around the boughs, being buffeted by the wind itself. The boughs are laden with leaves and seeds, and the sound waves are muffled at a turn, coming through at a turn, as this branch shakes and wags between the bells and me. The sun is filtered and flickered, I see the veins of leaves through. The insects buzz. One shield-like bug, as thin as a blade and about the size of a fingernail, lands and waddles around my blanket, its six legs hyper-optimized by evolution. There are perhaps 10,000 such creatures in this field. A trillion generations to produce it. It starts, its wings buzzing a thousand times a second. The hundreds of thousands of blades of grass wag in waves like grain, like carpet, like kelp in a fishtank, and they too, rustle. Could digital media capture this? I move my head to the side, angle it up, cup my ears, and I sense this field...I am absolutely surrounded by information. The amount information is incredible. I could hear the scrapes of the leaves against each other and the audio shadows of them as they blow around in front of this bell! I can hear the bells harmonics, and in my mind I see the sound waves racing round its circumference, overlaid with other harmonics making it jiggle like a waterballoon. The information there is 10^30...10^40. We can't even imagine.

And yet, we are told that VR and digital media will usher us into a wonderful world of entertainment. Pah! Lossless copies. Pah! We cannot even capture a millionth of something like that.

I think you can copy microfilm perfectly, as long as you know the smallest distinguishable feature which the original microfilm archiving process could create. You just have to sample the microfilm at twice that spatial frequency, by the Nyquist theorem, right?

That only covers the errors due to aliasing. There's still going to be additive noise introduced during the transfer that will build up each time the content is transferred.

Even sampling above the Nyquist rate, you can only reconstruct a signal perfectly if the samples themselves are perfectly precise - i.e., their binary or decimal representations have infinite digits. In practice the samples are truncated to a given sampling depth, and the difference is regarded as noise.

In practice though, I think you could take your idea even further into the realm of lossy compression. If you're scanning newsprint for instance, the vast majority of the page is either 1) text in a known font or 2) images rendered through halftoning, that is a regular pattern of circles wih some finite range of radii. Conceivably a system intelligent enough to recognise and encode this information might even be considered a higher fidelity scan than a simple photo, as it will repair incidental defects from the printing process.

I envision a transparent semi-crystalline substance that can be written to by a laser or two. With the volume of a sugar cube, perhaps, with the longevity of amber. Written down on the molecular level. Formatted to let 3D readers orient themselves to the playback, regardless of considerable distortion.

Those who've seen the 60s George Pal version of "Time Machine" will remember The Traveller spinning a disk on a flat surface to activate a voiced playback. Still visionary.