Comment by lizknope | Hacker News Reader

Comment by lizknope | Hacker News Reader

Parent Full thread

Tape doesn't last forever either.

https://en.wikipedia.org/wiki/Linear_Tape-Open#Generations

LTO-1 started in 2000 and the current LTO-9 spec is from 2021. But it only has backwards compatibility for 1 to 2 generations. You can't read an LTO-6 tape in an LTO-9 drive.

https://en.wikipedia.org/wiki/Sticky-shed_syndrome

> Sticky-shed syndrome is a condition created by the deterioration of the binders in a magnetic tape, which hold the ferric oxide magnetizable coating to its plastic carrier, or which hold the thinner back-coating on the outside of the tape.[1] This deterioration renders the tape unusable.

Stiction Reversal Treatment for Magnetic Tape Media

https://katalystdm.com/digital-transformation/tape-transcrip...

> Stiction can, in many cases, be reversed to a sufficient degree, allowing data to be recovered from previously unreadable tapes. This stiction reversal method involves heating tapes over a period of 24 or more hours at specific temperatures (depending on the brand of tape involved). This process hardens the binder and will provide a window of opportunity during which data recovery can be performed. The process is by no means a permanent cure nor is it effective on all brands of tape. Certain brands of tape (eg. Memorex Green- see picture below) respond very well to this treatment. Others such as Mira 1000 appear to be largely unaffected by it.

Data migration and periodic verification is the answer but it requires more money to hire people to actually do it.

I've got files from 1992 but I didn't just leave them on a 3.5" floppy disk. They have migrated from floppy disk -> hard drive -> PD phase change optical disk -> CD-R -> DVD-R -> back to hard drive

I verify all checksums twice a year and have 2 independent backups.

I have restored a few hundreds LTO-1 and 2 tapes using an LTO-3 drive a few years ago. If you keep the drives around and run Linux (which supports obsolete hardware better), keeping LTO tapes 10, 15, 20 years is not a problem at all.

A few weeks ago I wrote for a customer a restore utility for LTO-4/5/6 made with a now-deceased archival system from a deceased software company. Most of these tapes are up to 16 years old, have been kept in ordinary office cupboards, and work perfectly fine.

But you're right that archival isn't much about the media, but is a process. "Archive and forget" isn't the way.

While tape does not last forever, the LTO tapes are specified for at least 30 years.

The more serious problem is as you say that the older drives become obsolete. Even so, if you start using an up-to-date LTO format you can expect that suitable new tape drives will be available for buying at least 10 years in the future.

For HDDs, the most that you can hope is a lifetime of 5 years, if you buy the HDDs with the longest warranties.

I absolutely agree that the average tape will last longer than the average hard drive.

I've got 30 hard drives in use right now and at least 10 are older than 5 years. A few are over 7 years old. I've also had hard drives die in less than a week.

Even if the data is on tape I want to emphasize that the tape needs to be periodically read and verify that the data is still readable and correct. Assuming the data is stable for 30 years and you can just leave it there is a dumb idea unless you didn't care about the data in the first place.

The same is true for HDDs or for any other data storage devices.

I have stored data for more than 5 years on HDDs, but fortunately I have been careful to make a duplicate for all HDDs and I did not trust the error-correction codes used by the HDDs, so all files were stored with hashes of the content, for error detection.

On the HDDs on which data had been stored for many years, only seldom I did not see any error. Nevertheless, I have not lost any data, because the few errors never happened in the same place on both HDDs. The errors have been sometimes reported by the HDDs, but other times no errors were reported and nonetheless the files were corrupted, as detected by the content hashes and by the comparison with the corresponding good file that was on the other HDD.

I also make checksums for every file and verify twice a year. Over millions of files totalling 450TB I end up getting about 1 failed checksum every 2 years. If you are having more frequent checksum failures I would check for RAM errors first.

zfs and btrfs would do this automatically and have built in data scrub commands.

I have a checksum, uh, dream for lack of a better word, but I fear I lack the talent to pull it off. The problem with a checksum is that it only tells you that an error exists (hopefully), but it does not tell you where.

Imagine writing out your stream of bytes in a m x n grid. You could then make checksums for each 1 through m columns, and 1 through n rows. This results in an additional storage of (m + n) checksum bytes. A single error is localized as an intersection of the row and column with bad checksums. One could simply iterate through the other two hundred fifty-five possibilities and correct the issue. Two errors could give two situations. The most likely is four bad checksums (two columns, two rows) and you could again iterate. The less likely is three bad checksums because the two errors are in the same row or column.

I ran the math out for the data being rewritten as a volume and a hypervolume (four dimensions). I think the hypervolume was "too much" checksum, but the three-d version looked ... doable.

Someone smarter than I has probably already done this.

This sounds like par2 with extra/different steps but I don’t know much about this space.

https://en.wikipedia.org/wiki/Parchive

> and correct the issue

Just use RAR with recovery record.

Everyone here is talking about the life of either drives or and media.

The key point ought to be that data must be regenerated within a specified period to avoid bit rot (through loss of remanence).

No matter how many backups you have if they're all made at the same time then they should all be regenerated at the same time (and within the storage safety margin period).

I did some work for a place that had 30 year retention requirements for lots of data. (And indefinite for others)

Iirc, the goal was to turnover to new tape media every 8-10 years.

Reply on news.ycombinator.com