Comment by darkwater | Hacker News Reader

Comment by darkwater | Hacker News Reader

Totally anecdotal but just a month ago I opened a box from my last moving (ended 4 years ago...) and I found a CD-RW labeled "Backup 2001", which survived 5-6 relocations, even an international one. Well, I put it into an USB DVD player I have aaaand, it starts spinning and spinning and spinning, and nothing shows up in Nautilus. Looking at dmesg I see a few I/O errors on the device. Then, I try ddrescue [1], start extracting an image, leave it doing its work for a couple of hours and... magic! I have a perfectly mountable disk image!

I got to live again some >20 years old memories, pictures of younger me with my friends. stupid things you would save as a late nerd teen with access to the Internet and a ton of emails in Outlook Express format (guilty!) that I could not open yet. I guess the format is probably not super-hard to parse (most of the content is plain text actually) but since I'm here, if you know of something already existing to convert it to mbox/maildir it would be cool!

[1] https://www.gnu.org/software/ddrescue/

Even with ddrescue you might lose a few bits. You will only find out when accessing those specific files and bits.

For my photo history I prefer to keep PAR files next to it. On a running mechanical disc I haven't seen issues when running the repair. I do expect to see some errors in the future and as long as it stays under a certain percentage, things will be fixable.

I started doing this long time in the past, when harddisks were awful. I would often lose mp3 files due to damage. With mp3 you can easily hear it. With jpeg images you can often see damage as well.

Not much point, way too time consuming. I dumped hundreds of CDs/DVDs as most of them were unreadable. 20% of them might still good, but really no longer had that energy to savage them.

Could you explain the PAR file and why that would help?

PAR files are a form of Forward Error Correction (FEC) (implemented with Reed Solomon). They can be used to repair some percentage of damaged blocks in a file or set of files. The percentage is determined ahead of time when you generate the FEC (PAR files) and imposes the same amount of storage overhead (i.e., 15% redundancy costs 15% additional disk space with Reed-Solomon).

https://en.wikipedia.org/wiki/Error_correction_code#Forward_...

https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_cor...

CDs already use Reed Solomon, though. But yes, you can double it up.

33% parity, apparently.

https://en.m.wikipedia.org/wiki/Cross-interleaved_Reed–Solom...

If you pay attention to the ddrescue output it is pretty obvious when you are missing a few bytes here and there.

I have long since moved away from optical media, really for anything, but especially archival. Long term storage to me looks like an array of magnetic disks held together with ZFS.

I recently got rid of all my CD-Rs I'd use for archiving back in the day. Moved everything to a NAS and dumped CDs.

The earliest CDs I had were from 1994 or so, though most were from early 2000s. They were mostly cheapest CD-Rs you could buy back then, so I was expecting most of them be completely unusable.

But most of them still read perfectly. I had maybe half a dozen (out of hundreds) that were completely unreadable, mostly due to peeling, about a dozen that were functionally unreadable, because there were so many unreadable files I couldn't save anything complete out of them, and maybe 10% that had less than 10 unreadable files on them.

I was surprised. I was really expecting almost all the early ones to be dead.

My earliest ones still read. My later ones mostly don't. I'm guessing the difference is the earlier ones were recorded at 2x speed (fastest that existed) while the later ones were recorded much faster (probably 8x or 12x), which means less laser burn in time? Really not sure. Either that or the yamaha drive was just much better than the later model plextor.

Yamaha... anyone that makes both snowmobiles and guitars, is cool enough to make my cd writer.

(And bizarrely, they did seem to make excellent ones)

I recently sampled a CD-R from late 90s, mostly videos. I couldn't open a few files. Storage: inside home, temperature range from maybe 18 at night to 35 max. I'll give a try to ddrescue.

Same storage conditions here, probably never over 30C though, but I really never cared for its storage either. Give a try to ddrescue, it's pretty straightforward to use, I just ran it with defaults and it worked pretty well, but it has a ton of parameters if you want.

Now I have stored the whole disk image in the cloud (BackBlaze), let see in 2043 if I can still access it!

The problem with cloud storage is that it's a needy thing. You have to pay for it in periodic payments the whole time it exists. And it's also controlled by someone else.

I do really wish there to be a type of storage medium that I could record, store somewhere, and forget about. And then when I need it years later, it would still be perfectly readable. Optical discs are indeed kinda that, but their reliability is hit or miss.

Unless you plan on digging a hole in the forest, every type of storage is going to cost some type of upkeep.

But keeping a box of storage media in your closet doesn't cost you anything extra, as opposed to cloud storage.

How expensive is your housing per square foot? I suspect cloud storage is cheaper.

I don't know how common it is to own your living space in the US, but where I'm from, it is fairly common. And I don't think "how expensive per square foot" is applicable when you aren't renting.

But either way, you won't be paying more in rent and/or utilities for having an extra box in your closet.

Unless that one box is the one that causing you to buy/rent more space to store things. In general storage space is very lumpy - people have a fair amount free until they don't and need to get more space - when you get more space you tend to get a lot more. (Of course downsize/decluttering is also an option)

We use square meters in Denmark. The diameter of a CD/DVD is 120 mm, so you can fit 64 CD's per m² if you order them in a grid (more if you do not) A CD is 1.2 mm think. The ceiling height in my apartment is ca 2.7m, so I could fit 14400 CD's per m². If a DVD can store 10 GByte, that is ca 144 TByte.

The average rent per square meter is about $20/month (more in Copenhagen, less outside the big cities).

I suspect I can not store 144TByte in the cloud for $20/month.

You might argue that I do have that many DVD's. But then I do have many small corners and shelves that is not much use for anything else. And it would be even more unlikely to be able to store say 14 TByte in the cloud for $2/month.

Thanks for doing the math; you're right!

Google Archive storage is $0.0012/month (3x cheaper than Amazon Glacier, which surprised me). 144TB would be $173/month, so 9x cheaper to keep in your home.

I think your estimate of how many DVDs you can put in a given space is a bit optimistic: you're assuming no protection between discs and that a floor-to-ceiling tower is stable. If instead we imagine putting them in cases that double their thickness and putting collections of them in boxes that add 10% to the total height, though, it's 4x cheaper to keep them in your apartment. But if you use Blu-ray instead of DVD it's 20x cheaper!

I concur, there is no "fire and forget" solution comparable to what a physical medium gives you. And I could have probably avoided the CD-RW corruption by dumping the whole image on a new CD / DVD / USB storage every now and then.

I also have some external magnetic HDD that is now already ~10 years old, stored in the same conditions as the aforementioned CD, let see how it fares in 2032 :)

Chose CD/DVD as archival medium was one of the big mistakes I've ever made. Most CDs/DVDs failed just after around 10 years. HDDs seem to be much more reliable.

There are several models of DVD called “M-DISC” which have gold plating. They are advertised as lasting hundreds of years.

Get DVD-Rs with gold foil. "Archival grade"

https://www.amazon.com/Verbatim-DVD-R-4-7GB-UltraLife-Archiv...

A desktop CNC engraver can be had for $500. Stones are cheap.

Archaeologists have read engraved stones after thousands of years of neglect!

And how many did they fail to read?

The ones where the cavemen forgot to engrave checksum data.

Of the ones that were stored appropriately, virtually all are still readable.

Whether the format it was written in is still comprehensible could be a more pertinent question.

A fair number only fragments remain. Probably most, though I'm not an archaeologist.

You jest, but here's an idea: print a block of raised letters (like old typesetting) on a 3D resin printer (mirror the characters). Roll a thin slab of potter's clay, say 5"x 8". Press the text against the blank slab, fire the clay in a kiln, and voilà, your bio or other precious document will last like Babylonian clay tablets.

I might have to try this with my RSA public key.

You can also try readcd/readom. It's really helpful in reading badly damaged CD/CDRs.

One thing that’s missing is discussion of specific disc drives. Even in the early 2000s, some were known to be picky and some were flawless.

It could be a case of using the original drive to read the media, versus some cost-cut version that’s on the market these days.

Outlook may have used Windows structured storage back then -- but then again it could just be my imagination.

https://learn.microsoft.com/en-us/windows/win32/stg/structur...

e.g. https://www.mitec.cz/ssv.html

>Outlook Express format

http://fileformats.archiveteam.org/wiki/Outlook_Express_Data...

Nice link, thanks! With my searches I wasn't able to find it but this [1] looks promising. Let see how bit rot affects a 8 years old Linux release...

[1] https://sourceforge.net/projects/ol2mbox/

EDIT: mmh looks like it's now `libpst` and it's actually in the Debian/Ubuntu repos...

I've oddly noticed zero issues reading CD-R's going back to 1999 as recently as last year. I could certainly have some unnoticed bitrot but compressed archives have restored fine.

What's crazy impressive is this was a CD-RW, not a CD-R.

Isn't archiving on a HDD a more robust solution?

Depends on the quality of the archival media. Magnetic hard disks aren't without their own failure modes. The magnetic signals on the disk eventually fade causing errors. Motors and arms fail. Cosmic rays flip magnetic state. Or just stray strong magnets if you don't store it right.

Really good archival optical media isn't susceptible to the same cosmic ray it magnetic fade while reducing the chances of disc rot or delamination.

Stray strong magnets haven't been a problem for magnetic hard disks for 30 years. I don't think the fading of the magnetic domains on the disk is a problem over merely geological time scales either, though I'd be interested in finding out if I'm wrong. The mechanical parts are the big deal, and for recent disks (last 25 years) there are also concerns with onboard Flash or other EEPROM losing state.

Cosmic rays will eventually destroy anything that doesn't have an active repair process, including archival optical media, but the time scales are longer than when the Sun is predicted to swallow the Earth.

See also my comment yesterday: https://news.ycombinator.com/item?id=33116760

IIRC there was an import/conversion tool in Outlook 2003/2007.

Is there a good Windows option for DVD-R and RW recovery?

Reply on news.ycombinator.com