Optical storage – the future of long term data preservation (2015) [pdf] | Hacker News Reader

Optical storage – the future of long term data preservation (2015) [pdf]

snia.org

44 pointsdsego40 comments

Page 13 prompts some questions. 128GB BDXL is positioned before the document's publishing year of 2015, but doesn't appear to have actually been released until 2017. The multi-TB discs extrapolated out to present day seem to still be vaporware. Why did advances in high-capacity bit-rot-resistant optical media apparently stall?

Downloading 100gb games over the internet became a thing, and h264 and h265 were efficient enough to prevent consumer video needing anything much larger than blu-ray.

Sony ODA is a thing, and uses 11 600gb discs in a compact cartridge to give around 6tb per cartridge.

I do not expect large-capacity optical media to become available for consumer use, but A 100GB BDXL can archive only 6 minutes of 8K video at 60 fps from a RED, so I would assume high-end visual production would want something better than 100GB per disk.

Sony will happily sell you ODA at 6tb per cartridge for $$$$.

I think the video production folk use LTO and HDDs for long term storage.

Everyone who still cares would like multi-terabyte optical - but it’s not a big enough market anymore outside of long term corporate archiving (the ODA use case), and Sony probably doesn’t want to canabalize that market.

Sony sells decks and recorders for their high-capacity "Professional Disc" that can hold up to 128GB per disk. And then there is their cartridge to compete with LTO, the ODA Gen 3 that holds, IIRC, 11 discs with a total of 5.5 TB per cartridge. A bit less than LTO-7 (and half as much as LTO-8), but I don't think LTO tapes can survive being boiled in water.

BDXL M-Discs are available but I imagine Sony didn't see a need for bigger Blu-Ray disks so there aren't ant M-Disc equivalents.

What is the utility of cloud computing for archivists? Is it too expensive? It seems like the longevity is fairly proven at this point. If you put something in S3 in 2006, it's still there. GCS was launched over ten years ago, too. With a multi-cloud strategy you could probably make the case for durability, certainly as a 2nd or 3rd replica that just happens to be dramatically more convenient than optical discs.

Considering that many government records are now produced in the cloud anyway, it makes a certain sense. USGS elevation models for example are hosted on S3. Are they also on tape somewhere else?

> Are they also on tape somewhere else?

I can assure you they are and that there must be a number of copies available for use on any respectable supercomputer.

It just may be that the canonical source of the data is, officially, the cloud and that they release the data by uploading it.

It is far too expensive due to the power. Remember that you're not keeping one drive per unit of data going, but multiple drives for redundancy. Then there's the power used by the fans, power supply losses, and then the datacenter cooling costs...plus all the network infrastructure and so on. Some of those are pretty small but it adds up to a lot to keep, say, 6TB work of data online vs in a LTO cartridge sitting on a shelf.

Drive manufacturers, storage vendors, and cloud providers have been working on lowering opex/TCO for years. Bigger and bigger drives, helium which reduces aerodynamic drag, tiered storage where stuff is moved to slower and slower spindles the longer it goes without being accessed, for example...in some cases automatically getting pushed to tape. But you need systems to manage all that, systems that take power, have support contracts, and introduce a lot of complexity.

Eventually flash memory will get cheap enough - but flash degrades with time, too.

Eh, but the whole point of the cloud is if you mix a cold storage workload on part of a disk and an active workload on another part of it then the marginal cost of the cold workload in terms of all that stuff you mentioned is almost zero.

https://blog.dshr.org/2019/02/cloud-for-preservation.html

Archivists need access to this data decades, if not a century down the line. I trust the Internet Archive to be around somehow for much longer than S3.

Yep. Companies eventually die. Sometimes they're reborn, like IBM, AOL, and Microsoft, but never with the same priorities. The Internet Archive is almost as old as Amazon and archiving is its entire reason for being. The pendulum will swing back as it always has and send things off big [mainframes, clouds], and S3 will be sunset in favor of serving whatever comes next.

Never heard of M-Disc (defunct) til I read this, and from the Wikipedia page on M-Disc about Syylex for "more durable" optical media.

Whats the potentially longest lasting optical media available to consumers today?

If your media lasts a million years but all the machines that can read it vanish in twenty, where's the win? I assume the evolved rats or bunny rabbits that follow us may reverse engineer it at some point in their evolution and then just shake their fuzzy heads.

Edit: Note, in the early nineties, a lot of library got sold proprietary optical drives as archiving solutions and when those became pink elephants, they wrote learned papers about "you can never what medium will last"

There have been various ideas about putting decoding instruction in human-readable format at the beginning of the media so that people in the future could use a simple magnifying glass to read the info that would allow them to build a device to read the rest of the data. Nobody cares enough to actually use that stuff though.

"5G Optical Storage" is designed with this in mind. The first layer can be read with an optical microscope and polariser which contains instructions for building something to read the second layer.

You can always make new drives or new ways to read media.

It looks like the original M-Disc went broke but discs are still being made by Verbatim? That one looked more promising than other formats, at least it's readable by normal drives. Future terapixel cameras or scanners could probably decode a disc from an image, right?

Shiny discs need a laser in a dark, stable space with lots of cache and a purpose-built lens and sensor to read them reliably. I don't see a camera or a scanner having that reliability.

Every new pixel you want to render from a lens reveals more and more flaws. That's why older lenses often perform poorly on newer digital cameras: the designers didn't have to contend with 24, 50, 100... megapixel sensors. Even the 24 baseline already pushes what the film they make 4K remasters from can support and that's twice as big as what most film still cameras use.

I worked in optical storage in the 90s. Then and now, the storage technology of the future!

When the Compact Disc was released, the data capacity was way more than typical hard drives for many years. But by the time DVD rolled around, HDDs became denser and cheaper than optical storage could ever be.

I got some 25GB bluray to record family photos, I do cull rather ruthlessly though.

CDRs are most reliable way to backup important documents for several years.

Compared to microfilm? Surely not.

The joke might be "several" years.

Clay tablets are still the gold standard for durability ;-)

They will never forget Ea-nasir.

https://web.archive.org/web/20150310133210/http://anthropolo...

Engraved rock?

Mask ROMs are engraved rocks.

You want something cheap, inert, resilient, and unlikely to be repurposed for any other use.

QR codes are one of the first sort of barcodes to come to my mind for storage; but I'm reminded of how poorly they encode text compared to analog characters on a printed page. The printed letters are also a technology that is already known and self documenting. If you can understand the content there's no intermediate representation.

The pattern of entanglements on the surface of a black hole.

I had a box of attic junk that had about 50cd-r discs with Various bits of data. Mostly old Linux ISO's - I can tell you that over ten years in an attic wiped Sony discs to nothing, damaged Maxwell discs and made Memorex discs auto eject. Various floppies from 1990 or later stored in the same box read just fine... So that blew my mind.

What’s the attic like?

My cool dry Colorado basement (high temp is about 67) has been good and many disc from the 1990s are perfectly readable

Cannot agree more. I had around 300 CR-R/DVD-R and a couple dozen of DVD-RW disks, of various brands, mostly Sony, Verbatim, or Maxwell. Although those claimed to be able to sustain 10+ years, some even claimed to endure 50+ years, quite a few started to fade in 3 to 5 years and most of them failed when I last check after around 8 years. I might be able to recover a couple of them, but I would say 95% of them were long gone before I checked. It was quite a pity that some videos I archived on those disks were permanently lost. I knew those optical drives could easily become unreadable if the surface got scratched but I did not realize it could just fade away or I could have just keep those I don't want to lose on HDDs. I did not got a chance to check my floppies as my floppy drive got busted and there was really neither much point to get a new driver nor I could easily get one. Data on my HDDs is just fine, even after almost 15 years not been powered on.

Now I'm getting curios on how long my data can retain on a SSD without powering on it...

Depending on active and storage temperature could be anywhere from few weeks to a few years.

https://www.anandtech.com/show/9248/the-truth-about-ssd-data...

I had a few dozen CDRs from a couple different manufacturers that I used as backup discs that sat in an air-conditioned dark closet for about five years. None of them were readable without errors. Today none of them are readable at all.

Doesn't CD-Rs use organic materials unlike BDs?

I think the parent commenter is making a funny. CD-Rs were often unreadable within a few years even under ideal conditions. Put one out on your desk where sun from a window can reach it and you might not be able to read the disc in a few months.

Putting a label on a CD-R kills them extra fast.

Reply on news.ycombinator.com