Why the Future of Data Storage Is Still Magnetic Tape
spectrum.ieee.org
spectrum.ieee.org
Even more modern formats are difficult to copy forwards. I'm currently tidying out my old computer collection, copying off data from SCSI disks, 5.25" floppies, DD floppies, HD floppies, and CDs. My USB floppy drive can't read DD floppies.
The SCSI path requires copying to a PowerBook G3, removing the internal hard drive, putting it into a USB-PATA controller, and copying it to my laptop. I'll need to do the same for all the 2GB JAZ cartridges. I don't even have a ZIP disc reader, so I gave those two discs away to another collector so he can read them for me. The 5.25" floppies are the hardest - they don't mount even on old Macs, so I'll need to figure out how to set up ADTPro to copy them over serial from an Apple IIgs.
If you're in the area and want some old hardware/software/magazines/user manuals, please let me know!
https://www.reddit.com/r/VintageApple/comments/99223h/peters...
The lesson? If you want to keep the data, also keep the computer that can read it. And make backups.
Or just copy the data over the moment you get a new computer, instead of waiting for several decades. As long as you keep moving the data from generation N to generation N+1, there's no problem.
I have two 1TB external hard drives. The data was added over a few years. But now to copy 1TB over USB 2.0 is beyond painful.
Fortunately I can just disassemble the drives and connect them directly. But that's if I have an old system with IDE controllers.
If I can find an IDE to USB converter I'd be back to square one (unless it is USB3). And I don't know if there is such a thing as an IDE to SATA converter.
Some of my older thumb drives clock in at 4MB/sec. My Internet connection both download and upload is faster than that.
4. Be recoverable
How exactly do you extract the information after you drown it in resin?
So when you are ready to recover it, you can just break the resin shell to get at the paper stack inside.
Elaborate?
At the consumer level it makes way more sense to buy HDDs. My suggestion would be to pick up something like the EasyStore 8TB external HDD [0] (on sale right now at Best Buy for $160, will probably drop again soon). You can open it up, extract the drive, and drop it into your computer if you want internal storage.
If you want to reduce the risk of data loss you can also fill one up with an encrypted snapshot of your important things and ship it to a family member. Very important if you live somewhere that's at risk of being destroyed by natural disasters.
[0] https://www.bestbuy.com/site/wd-easystore-8tb-external-usb-3...
Plus you only do it once. Differential backup would take down the size of the next backups to a few Gb, or even Mb.
That's to say, there's more to it than upload speed, if Backlaze had servers in Europe I'd recommend it much more than I do.
I have several TB's of other data (mostly photos, videos, etc) that I'm fine waiting for weeks or months for if needed, but if you're in a hurry, Backblaze will sell you a hard drive that they restore your backup to and mail it to you.
Or I could just sign up for a cloud backup service, spend a month or so uploading my initial snapshot of data, and then backups are automatic and always offsite, replicated, and scrubbed. For $10/month.
https://www.amazon.com/HP-LTO-5-Ultrium-External-EH958B/dp/B...
The tapes are $31 and only store 1.5 T.
https://www.amazon.com/LTO5-Ultrium-1-5TB-3TB-Case/dp/B003KR...
I can buy a 1.5T hard drive for $47.
https://www.amazon.com/Generic-1-5TB-Internal-Desktop-Drive/...
I'm not seeing the cost-effectiveness of tape.
Harddrives usually last about 10 years without power at best before you loose significant amounts of data (you loose bits beforehand, don't worry).
If you can find a working drive, and drivers, for a 30 year old tape. Look at all the problems NASA has with their old tapes. The tapes actually are fine, it's just a major effort to custom build a machine to read them.
You can still obtain LTO-1 compatible drives nearly 2 decades after the standard was released.
The latest version of the LTO standard is LTO8 and drives are around $2700 http://www.backupworks.com/Quantum-LTO-8-SAS-tape-drive-TC-L...
Raw capacity of LTO-8 is 12TB at $160, which is a much better value than LTO5
Factor in that these tapes are designed to be written to and then sit on a shelf for years and that such devices are more frequently purchased by enterprises expecting a long service life out of them (say 10+ years) the prices for the complete system do tend reflect this with a higher cost to entry.
This is important. Your $47 consumer hard drive will likely not have that longevity, nor will your home computer 10 years hence likely have the correct interface and drivers to access it.
Ditto for my 5.25" floppy drives.
I finally threw away my old magtapes because there is no way to read them.
I threw away my zip disks for the same reason. And my MD disks. And my old tape cartridges from the 90s - no way to read them.
My old computers I retired in perfect working order will no longer boot (probably bad capacitors, who knows).
VHS and LaserDisc players are no longer made.
The only long term solution I've found is to buy new hard disks every year and copy the data forward.
And companies regularly have interns carry harddrives between servers or even have someone drive it between cities because it's simply cheaper to pay for gas and driver than to pay the transfer over wire. And usually faster too.
So that’s 10PB/4 days = 231 Gbps
Not bad!
- Tapes are easier to stack and load in boxes - Tapes are are more resistant to shock and damage from vibrations - Tapes are generally more resistant to damage from the environment - Tapes weigh less than hard drives (this is true whether measured per tape or per byte)
IMHO you should just account for the time to mount/connect the HDDs to the computer.
That is, HDDs are generally unbounded in the parallelism because disk bays are cheap [1] and plentiful.
OTOH, tape drives are expensive, multiple thousands of dollars, which makes them scarce.
[1] $100/bay in quantity, but, in some cases, effectively $0 if counting unused bays in existing servers.
Hell, you could even bring the drives with the tapes!
> Even with drives, if you get too elaborate of an enclosure or JBOD, you're looking at thousands too.
This actually supports my point (except the "too", which is misleading). One can reasonably expect that $2k-$3k enclosure to support 16-44 disks for that price [1], compared to a tape drive's singleton.
[1] Full-featured Synology NAS with 8 drives is under ~$1k, for example, while a CSE-847E1C-R1K28JBOD holds 44 drives for $2800.
But - this is a silly argument. I think we actually agree on all of this and we’re talking about an extreme hypothetical that while I’d love to test out, isn’t something I’m going to do in the near future!
Although, a few years ago, I did have a similar issue. I needed to love about 500TB of data from one data enter in Menlo Park to a new one in SF. We tried everything we could to make sufficient backups, but it was hard to backup 500TB of data over even a 10Gb/s link. We ended up just moving the physical JBODs in the back of a rental car. Most stressful drive I’ve ever made.
That's a good point, and it certainly improves the attractiveness of tape overall. Still, a $200 saving in medium doesn't make up for a $2600 difference in drive cost. How well does it improve the appeal of tape for parallelism?
The media+drive cost of LTO-8 for 12TB would be around $2850, while for disk is only $550. Considering LTO-8 has transfer speeds of 1.6x or so that of disks, that means only 5 tape units are needed to match 8 disks. That's still $14.2k compared to $4.4k, over 3x. (With disks, I'd want to do RAID6 or RAID-Z3 at the very least [1], which further reduces that ratio, but not below 2x).
That's certainly much closer to parity than an off-the-cuff estimate, but it's still not close enough to be practical. Anyone needing to transfer data in bulk, fast, would still do well to choose disks, not tape.
> this is a silly argument. I think we actually agree on all of this and we’re talking about an extreme hypothetical
I disagree as to silliness, and as to argument, since I also think we generally agree. This is a discussion in the spirit of HN, satisfying intellectual curiosity.
I also disagree that it represents an extreme hypothetical. I believe your own anecdote is nowhere near as rare as you imply. Even at lesser scale, it's an issue: the existence of the AWS Snowball is a testament to that.
[1] Ideally true sector-level ECC, though AFAIK, ZFS can provide a close enough approximation
Although one could consider HDDs to be immediately usable at the destination once they're "plugged in", there are too many exceptions (including good reasons not to re-use a device subjected to such conditions in production) just to assume it's true.
If they were SSDs, I might agree, but those are still too expensive for the use case.
I haven’t thought about SneakerNet since the late 80’s floppy disk days.
So really, the bottleneck is _mounting_ the tape.
It's all fun to say "oh a single tape can do 15TB" but it must cost in the thousands and thousands of dollars.
I remember when I was younger, I got a single SCSI external tape drive, and some tapes, from the Micro Center in Cincinnati Ohio. I then used that to back up my computer while I was in college. I've forgotten so many of the details, like the model of drive and tape, and the backup software (it was Mac OS). I do know it wasn't LTO, though, because it had to be rewound.
These days, were I to do something like that, I'd do BD-R. Yeah, you're only talking 50 GB or so (uncompressed) per disc, but that's not bad!
Linux even had a driver for it at some point (ftape), but IIRC it has since been removed due to lack of use. Never used it though, by the time I got more into Linux that drive was already obsolete.
But yeah, today the startup cost of a tape drive is so high that it doesn't make sense for home usage. At home I use borg backup (https://www.borgbackup.org/ ) nowadays, backing up to external USB hard drives.
Tried to use it on an old p3 box with win95, and was utterly surprised that win95 backup application had support for generic backup tape drives that allowed me to restore the tape content.
I also love the super smooth mechanical sounds of tape drives, even at the cost of slow seek.
https://en.wikipedia.org/wiki/ArVid
Danmere Backer https://www.youtube.com/watch?v=TUS0Zv2APjU (LGR)
But I'm really surprised that nobody[1] is offering "tape backup as a service". Pay $30 per 3TB cassette (slightly over market rate), a $10 loading fee and then $1/h for 100MB/s write access. When you're done, pay another $10 and it's shipped to you. Or $5/month for fireproof storage.
This might have to be a pre-booked service, or they could buffer upload onto spinning disks and write to tape after the fact (much faster), but for storing more than 1TB, this method could be vastly cheaper.
[1]: That one quick Google could find.
In the simplest version of my post, you're paying for media, the time for somebody to pop it in the machine, the hire of that machine while you copy data to it over the network, and the postage of a tiny little LTO back to you for archival.
On higher density tape, you could be looking at a one-off $100 total for writing 10TB of data to tape and mailing it back. There's a lot of room between that price and the nearest incumbent to make a strong profit there.
rsync.net seems very bespoke, so they may be able to give you a tape, but you're looking at $250 per "incident" plus hardware. They waive that fee if you store more than 100TB ($2k/month).
Yeah, they're much more available but most of us aren't after cloud storage, we want a lasting backup in case the NAS dies, or the house burns down. Restore speed doesn't matter.
I'm definitely not claiming to have all the answers here but there are points where tiering your storage strategy to get it onto something stable and without running costs makes sense.
10x that, or a petabyte, would cost $144k-$180k after 3 years, depending on region. Hardware [1] might be, generously, $60k. The hardware would also have operating costs of another $26k [2], but Glacier would have retrieval and data transfer costs if one ever wanted the data back.
Of course, tape, the subject of TFA, is even better. $17k [3] for the hardware and minimal operating cost in the labor required to change the tapes.
VC-funded startups have been tending to ignore this artithmetic (which applies to all cloud infrastructure, at only slightly larger scales), lately, so cost isn't always an issue any more.
[1] Let's say 100 12TB drives for 84 data, 12 (triple) parity, 4 warm spares. 3 enclosures, 6 SAS HBAs, plenty of CPU and RAM for ZFS.
[2] Generously, the above is 2.7kW. Adding 50% for cooling, at 12c/kWh. A commercial datacenter could double that (or triple or more, but, for backups, presumably one would find a geographically cheaper one).
[3] LTO-8 drives are $2700 and 12TB cartridge are $170.
> LTO-8 drives are $2700 and 12TB cartridge are $170.
What I initially ignored was the cost of a "library" (robot), which is, arguably, necessary if the minimum time to write a tape is over 9 hours (longer than a standard workday, meaning 1PB could take 17 workweeks to write with a single drive with no robot).
Including a library to hold all 84 cartridges might be tricky, but it appears there are 80-slot ones in the $12k range, which brings the total tape price up to $29k and reduces the write time to under 5 weeks.
Alternatively, one could just buy multiple tape drives and no robot. $25k gets faster write time than above.
In the world I support, everything has to be funded by something. We have our "cheap and deep" storage, for example (http://oak-storage.stanford.edu), but labs who use it have to pay for it.
Unchecked growth is highly unlikely, as grants run out eventually, and then the data will have to go somewhere else. Often these days, that ends up being Google Drive, but there is demand for tape archival storage.
Even the Stanford Digital Repository (https://library.stanford.edu/research/stanford-digital-repos...), who are extremely serious when they talk about "long-term", say they're developing pricing models.
It's tens of seconds, up to a minute, but that's just average, so I think we're on the same page here!
Even when not in academia, everything has to be funded by someone at some point. Sounds like you have a pretty awesome job though, and I imagine Stanford is one of the least likely to get cut off from precious NSF grants!
It was only after a year or so that I was cleared to visit the data centre and discovered that the mechanical tape silos had failed years ago. Every tape had to be located and loaded by hand. I felt guilty about all those datasets I had ordered on whim... and was surprised that the DC guys didn't have a 'Hit List' of progammers.
While we have a real roadmap, and a working solution of MAMR, much more so than the proposed HAMR from Seagate, that will scale us to 40TB per HDD by 2025, and may be more so given we don't really know the limits of MAMR yet, and we manage to put more platter inside a helium drives. Which I don't think is possible with HAMR, laser heating plate with helium filled drive has got to fun inside labs.
NAND Flash is also dropping in price, the current NAND spot price are already backed to 2016 price or may be even slightly lower. We have Fabs from Toshiba, Samsung and SK Hynix all coming online in 2019, all while 3D NAND are achieving better yields. The "Ruler" that Intel proposed has now standardises and becomes EDSFF, with up to 1PB storage in 1U. All current roadmap suggest we should hit 8PB or 16PB in 1U by 2025. In terms of 48 Drive in 4U server, HDD offers only 480TB per 1U by 2025. That is 16 to 32 times the density difference comparing to NAND.
In a system where information is stored in different drives like black blaze[1] , your reliability reaches a point ( I think, forgive me if I am wrong ) no different to those offered by Magnetic Type. What are the advantage left for using magnetic type? When cost, performance, space, reliability does no longer flavours them?
[1] https://www.backblaze.com/blog/cloud-storage-durability/
We have also not even reached any kind of limit on density. Sony has tapes that hold hundreds of terabytes.
So, they last forever, are extremely reliable, and extremely data dense.
HDDs exploit the same concept (magnetic storage) but trade speed for reliability. NAND is more reliable but still has a shelf life, and it also relies on support chips to constitute the overall media. It's also expensive as hell.
Tape is tape. You could smash a cassette and respool the tape and read it. Its also cheap as hell per gb.
Tape will be around for decades if not centuries to come. NAND is not a challenger whatsoever.
A key piece of the puzzle here is that not only are tapes used for backup, but to free up space from data center HDDs in order to increase their shelf life.
Stop thinking of tape as archaic. Magnetic storage is the foundation of modern technology and computing, and in past decades we barely scratched the surface of its potential.
We aren't talking about audio cassettes with analog audio recorded to them. We are talking about a storage medium that is more data dense than HDD or NAND could ever be and we have yet to reach any sort of limit.
With tape you can achieve more and more space by manipulating the width and length of the tape, the tape heads, the angle at which tape is read, and the way that the data is read and stored. It's also just a plastic cassette with polyester tape coated with rust particles. It's inherently cheaper, requires no circuitry, power, or motors within the media itself, and will last 50 years.
Tape is the purest implementation of magnetic storage concepts. As such it is going to improve faster than HDDs because there are no other design factors like platters, motors, and heads. If an HDD dies, you might be able to recover the data through expensive and time consuming procedures. If your tape drive stops working, you buy another drive and still have an intact tape with all of your data.
Additionally, while SSDs are wonderful, they are a pretty backwards design when it comes to efficiency. You are essentially trying to model magnetic storage by using integrated circuits. You are storing values with digital logic in a physical state on a chip.
Its fast and more reliable than a spinning metal disk, but it's expensive and not data dense at all.
The only threat to tape is public perception of it being "old" when it's really the most robust and best engineered data format we have.
I agree with the former, but I think your conclusion misses a critical piece of technology development: volume (which drives funding).
Of course, HDD volume is going down, while SSD volume is going up. That may well give tape enough of a comparitive advantage, if it hasn't already.
> The only threat to tape is public perception of it being "old"
For a definition of "public" narrowed a bit to include only/mostly computer professionals, I'd add "slow" as a mis-perception.
Other perceptions I, personally, have is that it's expensive and inflexible, mostly in the context of startup companies. I don't believe that's any kind of threat to tape, however, since it's likely these same companies will gladly over-pay for a cloud version of tape.
JFC. Harry get the key to the basement, need to dig out some tapes. San, can you pop down to the store and get some matches and lighter fluid...
Encryption decreases the volume of data that needs to be most carefully protected from the entire data set down to the key (and backups of said key). I imagine that data on mediums that make data retrieval and writing expensive can be encrypted with compartmentalized keys. So when Jane Doe wants her data deleted, delete the keys.
Keep a log of user ids who have requested deletion and purge their data whenever doing a restore. And don't keep backups for longer than required, which I think is generally something like five to seven years barring specific legal requirements.
Seems a bit odd to see such a blatant P.R. piece in IEEE Spectrum.
Wasn't optical going to eat rust-based cold data storage technologies? What happened?
I've heard it speculated that glacier's slow restore times is consistent with a large number of optical disks and robots for loading. Never heard it substantiated though.
https://panasonic.net/cns/archiver/
https://perspectives.mvdirona.com/2016/03/everspan-optical-c... (Sony Everspan dropped off the Internet; did they go out of business?)
In comparison to tape, it's not that much storage, but fairly cheap to implement at home. £60 for the drives, and about £15 per 100GB disc.
For VHS, yes, they are angled. 30 degrees, IIRC.
If that’s, indeed, what you’re asking.
It seems
"Intel on Tuesday introduced its new form-factor for server-class SSDs. The new "ruler" design is based on the in-development Enterprise & Datacenter Storage Form Factor (EDSFF), and is intended to enable server makers to install up to 1 PB of storage into 1U machines while supporting all enterprise-grade features."
ref: https://www.anandtech.com/show/11702/intel-introduces-new-ru...
Don't tapes have issues with print through and adhesion if not regularly used? I seem to recall best practices including something like annual seek to the end and seek to the beginning. Seems pretty similar to spinning disks.
I'm less sure of flash drives on a shelf.
https://www.ibm.com/support/knowledgecenter/en/STCMML8/com.i...
And as a bonus, while a SSD is only guaranteed to retain data without power for a year, tapes are designed for decades of cold storage.
Additionally, you could put up to 2x4x9 (HxWxD) tapes in there, at max capacity that is about 864TB/2.16PB
The only downside is you need another 1U to read the tapes, though the rest is automatic if you have a robotic tape library.
I'm having trouble finding where in the article that occurred.
To me, it seemed to ignore flash/SSD, as the principle premise was that tape survives due to its low cost (at scale), so only the highest-density HDDs were mentioned.
What exactly is this referring to?
Today, a modern tape cartridge can hold 15 terabytes. And a single robotic tape library can contain up to 278 petabytes of data.
Current 14TB disks are around 1000 gbit per square inch or 166 times denser.
Sure you need a motor, seek head, etc. All about the size of a tape. But you don't need a robot or tape vault either.
Seems unclear where the cross over for tape vs disk is. Especially when you use things like a backblaze storage pod and spin down disks when not needed to avoid the heat, power, and wear and tear.
Does make one wonder why not disk robots? The disks are the same size, support random access, and can be hot swapped.
I dug around and couldn't find a price on a 15TB tape, or even what it was called.
For long term storage, it's unlikely that disks will last as long as tapes.
All the electronic and mechanical components of a disk drive must still work in order for you to be able to retrieve data, and it's somewhat likely that if you just stick a drive on a shelf for ten years some percent won't be working afterwards.
For tapes, the separation of electronics from the actual storage medium itself means better maintainability; you can rpelace the tape drive without touching the cartridges.
Because a tape is relatively cheap, the drive is not. With a hard drive robot you lose the (small) cost of a SATA/SAS controller per drive, but you add the cost of the robot. If you're going to get disks you might as well just hook them up.
To be honest I never really understood the main reason for adding compression, especially since you can compress individual files far better using existing algorithms and then write those to tape. It must be a marketing thing.
The 10TB tape I found was "IBM 3592 JD Advanced Data Tape Cartridge 10TB/30TB (2727263)"
https://spectralogic.com/features/ts1155-technology-tape-dri...
Looks like tape as a service.
And that's not taking into account the extra money Amazon makes from the requests and data transfer that could offset storage costs.
I wouldn't be surprised if the glacier data was mixed right in with s3 storage. Just deprioritized when io is high.
I’d say this the same for any systems optimizing for availability over RPO.