Companies are once again storing data on tape, just in case | Hacker News Reader

Companies are once again storing data on tape, just in case

wsj.com

40 pointsfraqed52 comments

I didn't know they ever stopped storing data on tape. Tape to my knowledge has been pretty much in continuous use at big corps since their invention. Sure there was things like AWS glacier becoming possible alternatives but tapes are pretty entrenched.

I'd think most new startups do not have tape backup system, because they either export copies to a disk, or to services like S3 (then Glacier), since tape requires significant investment. I don't even believe SOX requires tape backup. Tape backup is common for storing master files like the master for a song. Those you can't risk keeping on a magnetic disk.

IMHO and from talking to a large number of people in startup leadership positions, many startups don’t even have a disaster recovery or business continuity plan let alone any formal backup process.

Sometimes they have expressed doubt that such things are even necessary since they are using distributed systems. In a lot of ways what I’ve seen in that regard is similar to the way some people confuse a RAID array with having solved their backup problem.

Your mileage may vary but this is common enough that not only is tape not part of it but even the backup concept is absent.

If you use cloud storage, your provider's tapes are your tape backups.

Where I work we have to have a tape backup. This is a business decision. If AWS is completely gone or someone accidentally deleted everything, we need to be able to restore our data in a few days and tape is the only option. This has been the business system for many years. That's how the music industry works. Someone at Pixar accidentally deleted Toy Story 2 scene...

To be fair, the second time was for creative reasons (they decided they'd made a bad movie and rewrote it). But you're right, the first time was an errant "rm -rf *".

Yep, i was also surprised by the article's premise. All the places I worked in managed nightly/weekly backups on ~220GB cartridge tapes. It was slow, but cheap and secure. Although I still have nightmares of having to manage Commvault's backup scripts.

The chart in the article essentially confirms your premise. It shows no decline; steady increase in tape shipments over time.

Some have speculated that AWS Glacier is just a service that still stores information on Tape anyways. I remember when I saw a robotic tape storage machine at Fermilab when the Tevatron was still active. That was awesome. They also had racks of tapes.

See https://en.wikipedia.org/wiki/Amazon_Glacier

Given that you have to request access to your data, then wait a bit of time before you can download lends credence to it being tape based, or at least something similar.

the power density of these data centers is so high that power and cooling may be your limiting factor.

In which case having racks of hard drives where most of them are powered off half of the time would be feasible. But then you have to schedule powering up the right racks for the current set of requests.

See Facebook's "Cold Storage" systems for an example of this

https://code.facebook.com/posts/1433093613662262

> To get the most data into this footprint, we needed high density, but we also needed to remain media-agnostic and low-frills. Using the theme that “less is more,” we started with the Open Vault OCP specification and made changes from there. The biggest change was allowing only one drive per tray to be powered on at a time. In fact, to ensure that a software bug doesn’t power all drives on by mistake and blow fuses in a data center, we updated the firmware in the drive controller to enforce this constraint. The machines could power up with no drives receiving any power whatsoever, leaving our software to then control their duty cycle.

> Not having to power all those drives at the same time gave us even more room to increase efficiency. We reduced the number of fans per storage node from six to four, reduced the number of power shelves from three to one, and even reduced the number of power supplies in the shelf from seven to five. These server changes then meant that we could reduce the number of Open Rack bus bars from three to one.

I remember that. It's like a drive carousel but without the arm.

I also recall years ago reading about someone who was modifying RAID controllers to spin up the drives one at a time on a cold boot, because peak current draw occurs while spinning up the drives. They were getting brownouts when an entire array spun up at exactly the same time. So they spun them up over a couple of seconds and everything was fine.

There's other speculation that Glacier is using high density Bluray optical recording with something like 400 GB per disc.

Isn't optical media a lot less stable than Tape? This seems like a bad idea if you're running an archival service.

I use to have a case of CDRs between 1997 and 2004, sometime in 2004 I noticed that many CDRs were becoming unreadable and switched to external HDDs. I kept the case for a while and I think it was 2005 or 2006 that all the disks were empty and I tossed it in the garbage.

Around that same time they were starting to phase out LPT ports on PCs and I decided to backup everything I had on reused tapes for an Irwin 40mb Tape drive from 1988 that I acquired sometime in the mid 90s from a salvage shop. I was able to pull everything off the tapes.

> Isn't optical media a lot less stable than Tape? This seems like a bad idea if you're running an archival service.

Unless you have some giant warehouse of multiple-copying disk-verifying and re-duplicating robots running around.. which in the case of amazon might actually be the case, and at scale, could save lots on media..

to be clear, I have no personal insight as to how their service works

That seems like a lot more trouble than it's worth.

From a cost perspective it makes no sense at all. An LTO-6 tape is $25 retail and holds 2.5TB of data, LTO-7 tapes are $90 for 6TB. A 4 Layer BD-R disk is about $15 retail and holds around 100GB of data.

I am surprised that this hasn't been leaked with how many people work at Amazon. Would expect some anonymous post somewhere "Oh yeah we use X, Y and Z there"

In all likelihood, this article is meant for people who don't worry/don't need to know about data storage.

Tape is perfect for things like police bodycam recordings. They need to store a ridiculous amount of data for decades, and don't need access to it in real time. Cops/Courts/etc are happy to wait a day for video retrieval because the costs are so much lower.

I've also thought about how you could store medical patient data on tape, and retrieve it the day before a patient comes in. Or storing other types of data, and having ML record what gets retrieved when and try to predict it to reduce retrieval times. Too bad other storage is so cheap!

Google uses tape. Very cool read/watch.

Summary: http://highscalability.com/blog/2014/2/3/how-google-backs-up...

Video: https://www.youtube.com/watch?v=eNliOm9NtCM

I've never set up a backup system that didn't have tape rotated into cold storage as its last step. As far as I know, nothing ever came along that offered the same value for longevity.

The suit is back! http://www.paulgraham.com/submarine.html

I don't think this is an example of a submarine article.

I use to work at a small health insurance company with a tech staff of 10 people. 1 persons entire job was to essentially managing the tape backups and getting them sent/picked up from offsite storage. This was a senior network administrator.

The primary advantage that tape still holds over disk is shock resistance. You can drop a tape cartridge and still expect to get all of the data off it, but that is not the case for a really high density disk drive.

An interesting thing that might change this would be one time programmable multi-level flash (OTP MLC flash). It has the shock resistance of tape, the random access of disk. The $/TB is slowly converging when you consider total cost of ownership and reliability of the equipment to read/write these.

At some point someone is going to do the needed mechanical engineering and that just might be the end of tape.

We still use tapes at my enterprise. I'm on the network team, so I don't really work with the tapes directly, but it doesn't seem as inconvenient as it might sound.

We have a few large "tape libraries" in our data centers. The tape libraries have a robotic arm that handles the bulk of the tape swapping and organization. Humans are involved in loading the tape library and carrying the tapes offsite for storage.

I'm speaking as someone who has had a couple of casual conversations about the tape libraries and am not involved in the management of them in any way. They could actually be a huge pain.

it's all about aligning the tape consumption with the size of the arrays/tape storage so groups of tapes need to be rotated at the same time.

if done properly, it's as simple as grab this 'row' in the robot and replace it with this 'row' from the storage area - when not, it becomes a confusing hodgepodge of shelves and containers and which-tape-is-where.

I don't think it ever went away in larger companies.

A few months back I was looking into tape hardware, and I was disappointed to find that the barrier of entry is too high for consumers with only dozens of TBs. Based on my calculations at the time, for my desired data size it ended up cheaper to just buy a bunch of HDDs.

How do HDDs compare to tape when it comes to long-term storage? I've read lots of mixed reports.

> I don't think it ever went away in larger companies.

Or smart companies.

heck, I evaluated tape storage for personal use. It cost way less than I expected.

We stopped using tape just because datasets are too large. Fulls and differentials are not feasible at petascale. With LTO7 - a petabyte takes over a month to backup with a single drive. You need 50 or so to pull it in a day, assuming your storage can do 15GB/sec sustained.

50 is not a large number. What's the problem?

50 tape backup drives is not a large number for less than a single rack of storage? Buy one rack of storage and two racks of tape backup for it? That seems... not great.

Even outside of managing hundreds and hundreds of tapes.

I think he means that you'd need 50 tape drives with the backup running simultaneously across all of them to run the backup in a day - fortunately that's not how backups get run, you generally aren't running full backups every time, you run a full then do incremental or differential backups starting from that point with periodic full backups.

The point I caught from the article that most people are bypassing is that tape provides a fully disconnected and unreachable (by attackers) backup, where many online backup systems are using software and systems that may be reachable from the systems being backed up. In the worst case of a motivated knowledgeable skilled attacker that could mean an attacker having the ability to wipe backups as well as live systems.

This is a good point. It really depends on so many factors what a good solution is for one's dataset.

What do you use instead?

We have fulls in the cloud - and the initial seed took forever but we have a big pipe that made it feasible. And then do file system diffs, checksumed with an index file. An object store would probably be a better solution for those without a giant pipe. With an aggressive HSM policy of moving data to it. But there are so many factors in terms of what sort of data you have and what you're doing with it, change rate etc. Our data is very WORSE (write once and read seldom if ever). So use tends to die off pretty quickly after 6 months or so which allows us to keep most data on near-line indefinitely.

"We have fulls in the cloud - and the initial seed took forever but we have a big pipe that made it feasible."

We (rsync.net) now have multiple customers that have allocated 1+ PB of storage and are using our 10gb connections[1] for the initial sync.

It's 1.2[2] cents per GB, per month but it's fully online, live, browseable storage - not a nearline/offline solution like glacier/nearline.

We'd be happy to serve you ...

[1] We use Hurricane Electric (he.net) for our connectivity in Denver, Fremont and Hong Kong.

[2] and dropping ...

Hey Rsync Folks, We've actually looked at you but we are a non-profit and something like Oracle being $0.001/GB offsets the offlineyness of it because costs rule for us. But, love your service and recommend it to those that have a better fit than us.

1.2 cents per GB per month sounds great.

Unfortunately http://rsync.net/pricing.html cheapest rate is 4 center per GB per month, and that's only for over 10TB.

People can and do use tape-backed archivers for multipetabyte datasets-I'm one of them.

HPSS was pretty mach made for this, although it is not a "backup software" in the same way as Networker, commvault, bacula etc.

http://www.hpss-collaboration.org/

Yeah, I think there is still a play for LTFS and using tape for datasets, but for classic backup at scale? Seems untenable.

A relative of mine works for quantuam, a tape producer. There is some interesting history about standardization and the big players ganging up on the little guys.

You can also buy tape drivees online :) https://www.newegg.com/Backup-Drives/SubCategory/ID-46

http://www.quantum.com/

How well does tape compare to DNA?

https://en.wikipedia.org/wiki/DNA_digital_data_storage

I'm not sure if you're kidding of not but...

Our ability to write DNA is quite limited (single fragments of ~1000bp at most) and expensive. The largest completely synthetic fragments that have been made would be able to store a few Megabytes of information.

I think it's potentially kind of interesting as an archival format, but even then data retrieval is always likely to be somewhat problematic. And we're a long way off synthesis of large datasets being practical.

Oligo size isn't all that important since you geneally synthesize pools.

You're right that reading the data back is expensive (essentially a sequencing project), but for deep archives that you would only ever retrieve in a catastrophy, DNA has the potential to be an ideal medium.

It makes it... less than ideal. And I don't believe we have good massively parallel synthesis approaches currently. Overall writing is problematic (reading also, but is getting cheaper/easier).

The length is not an issue since you can sequence en masse, and assemble the results by searching for overlapping regions.

However the expense of the reading and writing machine and the reagents, and the pain of sequencing are going to be prohibitive. There will almost assuredly always be better and faster, and cheaper direct ways of manipulating electrons than molecules.

Is it easy to accidentally erase/corrupt tapes with magnetic fields and such?

IME the bigger problem is the tape reader failing and the replacement not being able to read the old tapes...

You can grab virtually any old type of old tape reader over on ebay. As long as the host system has drivers for it (IIRC even 8 years ago there were no mainline drivers for Iomega ZIP floppies), you'll be fine.

And for really crucial data, you can always go to a data lab and have them restore the tape; wasn't there a story a year ago or so when they found old NASA tapes?

A standard tape horror story is that the old drive ended up out of alignment and later died, then the replacement, same type, was aligned properly (or just differently) and couldn't read the existing tapes.

Reply on news.ycombinator.com