The Reliability of Optical Disks
blog.ligos.net
blog.ligos.net
I burned hundreds of discs a decade ago and read them all back perfectly, to my surprise. I strongly agree with the write-once behavior for photos and downloads. HDDs worry me in the long term because even if the platter holding the data is fine, the R/W head, mechanics, and electronics could fail, not to mention that a head crash could wipe out good data.
I'm extremely disappointed that the capacity of optical disc formats are essentially frozen, topping out at 100 or 128 GB for BD XL. They're definitely not keeping up with the continual increase in storage density and decreasing cost of hard disks and flash memory. Why are new formats not forthcoming? Where's my 300 GB Archival Disc? Where's my 1 TB holographic disc? Where's Microsoft's glass optical disc that's stable for millions of years?
I want to keep on burning optical discs and putting them in cold storage, but this gradually obsolete technology is not making a convincing case for me today.
Previously: https://news.ycombinator.com/item?id=26038893 , https://news.ycombinator.com/item?id=29221418
If you want huge local offline storage, the solution is tapes.
LTO tapes do store the required amount of data and if you want them to last extremely long you should make new copies every 10 years or so. Just like you should for optical media. Sure, some discs might last 50 years. But some others won’t be readable after 15.
The selling point now isn't that is a temporary storage solution that happens to last long like CDs and DVDs. I'm sure you have to make a pretty good case to get a customer to switch their high reliability data storage to a novel format.
I do hope it can eventually come back though.
My OS (Win10) is on the same SSD. When I made the decision to get a 1 TB SSD for the OS and media storage, deep down a voice whispered [dont do that]. That decision has come to haunt me 1 year (!) later. I used to use 128 or 256 GB SSDs for the OS and keep all my media on another HDD drive (which failed after 10 years).
Also: 1 TB discs:
https://techgeekforever.wordpress.com/2014/03/11/sony-and-pa...
reads the linked comment
notices my own reply
...oh. Nevermind.
So, I'm going to go out on a limb here and say that all of the new formats are not "forthcoming" because of basically two problems that happened at roughly the same time:
- Newer optical media formats became too expensive to implement in consumer devices
- Optical media is inherently user-hostile, almost as much as tape
Ok, first claim first. Back when Blu-Ray first came out, drives were hideously expensive; to the point where the PS3 was basically eating SONY alive because the drive itself cost as much to make as the whole console sold for. Apple outright said they'd never ship a Blu-Ray drive in a Mac because it was too expensive and the patent licensing was a nightmare. Even today Blu-Ray drives are still more than twice as expensive as DVD drives.
But, of course, this isn't the only hurdle, but it did dramatically slow Blu-Ray adoption. The spat between HD-DVD and Blu-Ray almost certainly didn't help either. All of that ultimately meant that people weren't buying Blu-Ray players and putting them in their living rooms; which meant that it didn't make sense to have one in your PC either.
The dirty secret of optical media is that it's remarkably user-hostile as a consumer storage medium. Discs want to be written from beginning to end, and preferably all-at-once with a single table of contents. This makes it very difficult to actually provide a drag-and-drop utility for getting files onto a disc. You can do this, but in most cases this is a hack involving multi-session filesystems that relies on readers knowing how to stitch together the sessions. This is why Windows had that entirely unhelpful "What do you want to do with this disc" dialog for optical media. If you try to use a Blu-Ray disc like a flash drive you will be extremely disappointed in it.
In the days of CDs and DVDs this didn't matter. The consumer benefit was that the discs you were making could go straight into a CD player or DVD player that you already had; the fact that you had to jump through some hoops to do it didn't matter. And if you bought CDs or DVDs you can play them on your computer, too. But far fewer people owned Blu-Ray players, and if they did the player probably also took USB drives. So there would be no reason to ever burn a disc to get a movie onto a player, and getting data onto them was enough of a pain in people's arses that they'd just use USB drives instead.
So with all that taken together, nobody is bothering to scale optical discs for consumers because there really isn't a market for them as a data storage medium anymore. The things that discs still do well - i.e. selling you prerecorded movies - also just so happens to still be fine with topping out at 100GB/disc, even for 4K video. Some games are starting to push beyond what a BD-XL can hold, but those games also are "live service" games that update way too frequently for "getting all the assets on the disc" to make any sense. All the supra-Blu-Ray optical data storage formats are all targeted towards the archival market because it's something that optical just accidentally does very well.
Are SSDs any better in that regard?
oooh I can answer this one. About 15 years ago I saw a talk from a team at, oxford I think?, that had invented a way to store many terabytes on an ordinary blue ray by adding rotation to the scratches, which can then be read because scratches at different angles polarized the returning light differently.
Anyway, they had a working prototype and everything and the big companies like Panasonic all said "No sorry, everyone is shutting down their optical drive lines. We have determined that the future is internet storage and solid state. No one wants disk no matter how much storage they can hold." And that was that.
My own experience is pretty good, but not 100%. I burned a lot of CDs from 2001 to 2006 or so. Most were discarded at some point, but in late 2019 I ripped all that I had (39 CDs) with (GNU) ddrescue. The vast majority (77%) had some small amount of read errors. I was able to recover 99.88% on the worst disk (a brand I'm not familiar with: "D"). That disk was about 15 years old. That's pretty good, but not 100%.
This experience underlined to me the importance of having parity files in addition to the disks themselves. When I do a Blu-ray backup now (roughly once a year for the most important files), I also burn a DVD with par2 files.
Back in the 00s I didn't verify checksums, but as a test I checked zip, gz, and png files on the oldest disk I had (Jan. 2001, 100% recovered). Those files have internal checksums to detect corruption. All those files passed. I should probably perform the same check on the other disks.
Edit: I ran my script to check internal checksums on my disk image for my worst disk and 19 out of 635 files tested were corrupt, all png files. That is worse than the recovery percentage that ddrescue reports (97.01% valid vs. 99.88% recovered). That's roughly 0.2% lost per year.
All my CDs/DVDs burned from about '97 onwards were in perfect condition. I think one deteriorated, stuff I didn't care about given to me. I never bought super-cheap ones, maybe that helped.
I did save a few "mix tape" discs from the era for nostalgia purposes. Imagine 15 of the best songs from different artists on ONE disc! It was a big kick to make your own CDs at the time... would blow people's minds, when you showed them. :-D
I just grabbed one from my DJ days: TDK dated 1997-01-01:
Listening to it now. We are… family ♪♫♪
EDIT: btw ofc all my pressed audio CDs all work just fine. My car for whatever reason, although semi-recent, still has a CD reader and I was listening to some Ennio Morricone CD in the car 10 minutes ago!
Multiple copies, multiple formats, multiple locations, is the key to back up.
That was 9 years ago but I still wish I had a backup of those mix CDs.
Anyway, I haven’t trusted optical since then.
In particular, you can do cost-efficient archives to multiple tapes and store them in safe locations. If (and that's a big if) the base cost of the drive is tolerable.
There was recently a wonderful post and discussion on LTO for "normal" nerds [1].
But that probably depends on the location you live in!
As wtetzner said, it's unlikely that a single event could cause all of these to fail. An EMP attack might take out my magnetic hard drives and SSDs but leave the optical disks unaffected. I've also had a hard drive fail due to impact damage that likely would not affect a SSD. Flooding is not something I've thought about before, but since my backups are located in different geographical areas, I should be covered there. Etc.
First of all, the magnetic gauss needed to flip bits on LTO media is huge. It's not a practical concern unless you're choosing an incredibly inappropriate storage location.
Second, why wouldn't you store your tapes in a case or a vault that is magnetically shielded?
You probably store your discs in cases, instead of insisting they be naturally durable against rough handling and abrasion.
You're right, it does seem huge: https://superuser.com/a/568367/111650
(Some of the links from this answer are quite interesting.)
The issue is that I don't know (and I suspect it's difficult to know or even classified) whether an EMP attack (the main scenario I'm worried about, which can be quite powerful) would be sufficient to cause damage. I don't believe there is a consensus on this though I'd be glad to be proved wrong here.
> why wouldn't you store your tapes in a case or a vault that is magnetically shielded?
I would; as I said I put some magnetic hard drives in Faraday cages. However, I'm not confident that the Faraday cage would be sufficient. A lot of the discussion online about constructing Faraday cages to protect electronics is speculation. I haven't seen any clear data showing that during an EMP attack magnetic data is protected inside of a Faraday cage, or protected without one.
Faraday cages are not intended for that purpose.
Magnetic shielding is a well understood concept. What you need is some thickness of ferrous metals.
https://en.wikipedia.org/wiki/Electromagnetic_shielding#Magn...
https://web.archive.org/web/20070327130322/https://advancema...
https://interferencetechnology.com/magnetic-shielding-basics...
It's easy to test the efficacy of magnetic shielding yourself if you have a large, powerful magnet and a CRT monitor handy.
I don't see that kind of EMP erasing magnetic media.
But on the flip side, the LTO spec guarantees that this two-generation backwards compatibility will remain for the forseeable future. As the article mentions, it's possible that optical discs will become unpopular and cease to exist completely.
With tapes, you're in the same boat as a bucket full of institutions to whom archives are VERY important. That's a big plus!
https://www.theregister.com/2019/05/31/lto_patent_case_hits_...
When I ripped CDs I had from the 00s in 2019, I used an external Blu-ray drive from probably 2018 or so. Seemed to work okay; see [0]. I didn't have access to anything older so I can't compare but I think both new and old drives are probably good.
The more I've been thinking about it, was it one of the audio rippers?
Perhaps the same is true of optical media. Are higher density optical disks from later in the era less robust than earlier tech.
Like the way Hobart mixers from the early 1900's are built like tanks and can outlast ones built today to save money.
https://www.vice.com/en/article/mg9pdv/the-hidden-phenomenon... [1]
https://www.howtogeek.com/682807/the-cds-you-burned-are-goin... [2]
Those are, in order:
1. Microfilm/microfiche
2. Ink on 100% cotton paper, or ink on vellum
The properties of microfilm have been well studied. It requires no technology other than a magnifying glass to read, has a well over 500 year lifespan in studies, and has exemplars that have survived more than 150 years.
Ink on paper, on the other hand, we have exemplars that are almost 1000 year old, with ink on vellum more than 2000 years old. It requires no tools whatsoever to read, and we have been working and understanding how it is best preserved for hundreds of years.
There are services that can do that for you. You send them document images, they send microfilm back. I've never actually done it, but I've considered it, an I had this random service in my bookmarks: https://overnight-scanning.eu/microfilming-service/
Drive: https://www.bhphotovideo.com/c/product/1549175-REG/sony_odsd... (only about $9k)
UPD. just searched, and found, that I needed to enable periodic scrubbing in Task Scheduler: Microsoft -> Windows -> Data Integrity Scan section. I already had data integrity streams enabled (another thing you have to do manually after creating the volume).
Also, we’re a quality over quantity family. We don’t want thousands of movies on several subscriptions. Everything we’ve ever liked fits on two four-foot-wide shelves.
That said, there are a lot more instances of "disk" than "disc" used in the comments here so far.
Apple—of all companies—has a support article on this: https://support.apple.com/en-us/HT201697
https://en.m.wikipedia.org/wiki/Spelling_of_disc
> the spelling disk is more popular in American English, while the spelling disc is more popular in British English
For anybody with deep knowledge of the industry, is the price of discs an accurate reflection of costs or is it inflated by secondary factors such as attempts to make the media more expensive to deter piracy or Sony's desire for high profit margins on data storage? I well remember Sony's shenanigans from the 2000s when they made great hardware and then only accepted MemoryStick instead of SD cards while charging much more for their flash format. I was so excited for BluRay and then prices just never came down enough to be practical.
As an aside, I'm of the impression not enough people know about this levy. It feels to me like the kind of thing that fouls up the "nothing to hide" arguments regarding privacy; that is, a law few know about, but one that affects lots of people that wouldn't necessarily agree with being charged for something they may not be doing.
While I'm sure some people still pirate and rip to BluRay, I'd assume they're just as likely used for things such as home movies, format shifting (which is legal in Canada [1]), or just plain backups (which I use for important files like a keepass database or 2FA codes).
While this has drifted off-course slightly, it may at least have given you an answer regarding why BluRay discs didn't fall too far in price - at least if you're based in Finland.
[0] https://en.wikipedia.org/wiki/Private_copying_levy#Australia
What do you do?
Write it to RO media and take it to different computers to verify the checksum/signature. You don't need to worry about other machines, which are also potentially compromised, corrupting the data. You also don't have to worry about something like BadUSB, where the USB device's microprocessor firmware infects the host machine (or vice versa) before the data payload is even read. This means you can put the disk in your secure airgapped machine at home, even after it's been in the machine at the local library. You create a concensus of file integrity after reading and verifying the data across a range of machines.
Obviously there are other attack vectors, but with the cost of these sorts of attacks (badUSB, or simply getting rooted) going down, the accessibility going up, and the potential rewards becoming more valuable and widespread, it's an attack vector to consider for important data.
About 10-15 years ago, it dawned on me that I hadn't seen a working tape drive in quite a while. Any tapes, even if the media were intact (highly doubtful, and the point of the parent article) would not be readable.
Much the same comment for boxes of punch cards. (My late colleague used to tear off pieces of the cards and use them for "filters" in his roll-your-own cigarettes.)
I guess my point is, even if the media are reliable (holes in cardboard? pretty foolproof from an ink-on-vellum viewpoint) the entire surrounding ecosystem needs to survive for archival purposes.
After dealing with several valuable SEGA Saturn games that succumbed to bit rot, and are therefore worth practically nothing - (nor are they playable, obviously) - I backed up whatever optical disks I had left and have never used optical media since. I certainly never will again.
I just can’t trust it anymore as an investment.
I’ve got vinyl from the 50’s/60’s that still play great but after losing those SEGA games I just see those little plastic disks as ticking time bombs.
Taking storage overhead (cases of some sort; spindles aren't well-suited to store BD long-tem) into account, the Chinese 20ct/50GB (plus patent; iirc sub-linear scaling with capacity?) option of a DL BD-R may easily be cheaper over all.
There's an open space for cheap tape drives.
...and not describing how they tested for errors...
...and not having particularly rigorous testing methodology (the DVD was apparently left for three months before he got around to checking it)...
...no organization looks at a requirement like "archive these records for 50 years" and then commands its IT department to only store the data on media that will be expected to survive 50 years.
What you do is store the data in such a way that your chance of losing data per year is within acceptable margins, on the most suitable storage method at the time, and migrated periodically when that system's chance of data loss falls below acceptable margins, appears to be getting too impractical to operate/maintain, or there are improvements in reliability or practicality.
That can include factors like cost, availability of knowledgeable labor (ask the IRS how finding COBOL programmers is going), parts/service availability/cost, and so on.
Other comments:
> HDDs, especially NAS disks, have an “always on” assumption
Even in the early 2000's, commercial NASes were offering systems that could power down portions of the array that were unused. There was significant interest in minimizing opex, and that's mostly power. Nowadays unraid and other solutions allow for the same.
> And if you ever wanted to take disks offline and store them on a shelf, you don’t really know how long they’ll survive - unless you plug them in every now and then.
"you don't know if that HDD is gonna work if it's sitting on a shelf" is irrelevant because your backups shouldn't be sitting on a shelf for any significant period of time, regardless of the media/mechanism. See discussion above: you should be rotating your media off-site periodically, bringing it back for tests. The biggest problem with HDDs is that SATA connectors are not rated for frequent use; their connect/disconnect rating is often in the range of "hundred or so." (This can be partially addressed by using a cable that stays plugged into the drive, and replacing the cable when it is past a certain number of cycles.)
> if you ever wanted to take disks offline and store them on a shelf, you don’t really know how long they’ll survive - unless you plug them in every now and then.
> And yet, the expectation is, that you will be able to read the disks without problem - even with zero maintenance. Indeed, that was the outcome of my ~15 year experiment!
That's not the expectation, at all? CD-Rs used to have a lifetime measured in months before they start showing failures, especially if they had an adhesive label. I don't understand how the author possibly could be reading ten year old CD-Rs. That's basically impossible.
DVD-Rs faired a bit better, and bluray disks even better.
The issue with HDDs is the lubrication of the motor
> any NAS will automatically check for errors, and notify if problems are found.
No. No. No. NO. This is probably one of the biggest myths of RAID and NAS devices.
Just because you have a RAID array doesn't mean it is configured to scrub its arrays; this often has to be enabled.
Just because you have a RAID array scrubbing doesn't mean you're going to find out about it. Make sure your reporting works, and ideally report on success, not just failure, and have something else that reports on failure to receive
Last but not least:
Just because your RAID array controller (or software) finds a parity error or unreadable block doesn't mean it does what you expect, and most people expect "beep bop boop, parity/mirror inconsistency detected. Fixed, meatsack! Pat yourself on the back for being smart in using RAID."
Reality: "beep bop boop, parity or mirror inconsistency detected, so go and verify your files or restore from backup, meatsack."
RAID arrays cannot self-repair mirror or parity errors because there is no way to do so. The array controller has no idea whether it's the parity bit that is wrong, or the data. In a mirror, it has no idea which drive is correct - just that one of them is bad. RAID is not for data consistency.
This is why ZFS is usually the top pick for data consistency: it knows the checksum of files and metadata, in addition to the on-disk redundancy via mirrored copies or parity. If it finds an error, it can play out both scenarios to see which results in files or metadata that match their checksum.
The cyanine dye CD-Rs were the most unreliable but the later phthalocyanine and azo discs are more robust. Mine all readback fine so long as they have been stored well without sunlight exposure.
Unfortunately, depending on your local mail setup and your user account setup, you won't get the mail that is sent on errors when it's delivered to the root account.
Many CDs and later DVDs. The DVDs were all seemingly fine but a sizable portion, maybe 20%, of the CDs are flaking along the outside. I was honestly kind of surprised.
Ideally there should be some machine to go through old backups and upon first detection of an error, then it could regenerate the array from the remaining discs of that array.