Hacker News Reader

Comment by h2odragon | Hacker News Reader

Once the latency requirement goes past ... i dunno, a few seconds? "Insh'allah" time frames; tape comes in and stomps all over the competition with its space / cost tradeoff. Dismountable media and a mechanism to swap tapes (even a meatsicle if need be) just has too many advantages to compete with.

Gets me thinking tho: Why stop at 5.25in drives? Let's go back to washing machine sized units or even bigger. Let's stack platters so dense and heavy that we can use the spindle as a flywheel UPS system too.

Or how about making a truly gigantic tape spool? Perhaps kevlar threads impregnated with iron could serve as "tape" and we could spin kilometers of that onto a spool.

View on HN

Tape also has some inherent disadvantages. Perhaps surprisingly, the lack of near-instantaneous seek to the data you want isn't one of them. The larger issue at scale is that it's not disk. It's a separate kind of device, with its own power and (often very stringent) environmental requirements, and its own software. Also big silos aren't cheap, both because of inherent cost and for lack of significant competition. Hyperscalers hate that type of heterogeneity, so they'll only go down that road if they feel they have no other choice. Yes, media are cheap, but the extra operational complexity can obliterate that advantage.

A lot of this doesn't apply as you go smaller, but also the drive cost starts to dominate (can't be amortized over many cheap tapes). There's a pretty narrow band where tape is likely to have a real advantage over super-high-capacity disks.

If you're actually hyperscale, say 100 EB of tape, then you'll have dedicated teams just for tape archives. The heterogeneity doesn't seem like much of a problem when you have 200 engineers x 300k USD per engineer to spend on managing the system.

If anything, we're seeing hyperscalers becoming more heterogeneous with CPUs. Google has training and inference TPUs, custom silicon for encoding media, and many more custom CPUs. It makes sense for storage if the benefit is there.

Those are completely different levels of heterogeneity. A GPU/TPU might draw a lot of power, but does it have to be separate power? Do you need to put that stuff on a separate slab to isolate it from vibration? Does it need its own air filtration system? Do you need a new runbook, because your usual redundancy approaches can't be used for million-dollar systems? Do your DC techs need new training, because the usual "yank and replace" isn't applicable either?

I've watched colleagues in a team adjacent to my own work through these issues. Have you? Slapping a commodity card into a box and loading some commodity software on it seems like a cakewalk by comparison. Obviously people thought it was worth it anyway, but I think you're seriously misunderstanding where the difficulties lie and what kind of resources are needed to overcome them. 200+ engineers would be overkill (unless FB engineers are ~4x more productive than Google engineers) but you'll need other kinds of specialists as well and the installation/operational costs will still be high. It's unlikely to be worth it for an organization much smaller or different.

Yes, these factors would probably limit massive tape archives to only the biggest companies.

Why can't tape be self-contained like HDDs?

Aside from the fact that most people using tape for archival storage don't want to pay extra for the read/write heads, SATA interface, etc., there is no reason why you couldn't package all these things into a self-contained tape unit with a small flash disk acting as a small cache and directory listing.

You could definitely package such a thing for consumers, for example, but most workloads there aren't a great fit for the medium. Basically the only thing that makes sense is using it for archival and backups.

> Why can't tape be self-contained like HDDs?

Because that gets rid of the main advantage of tape. Which is that the tape-media has no read/write head and is therefore much much much cheaper to mass produce.

---------

In practice, people buy tape-libraries entirely. Like a 3U unit with 50-tape slots + a few drives to read/write to those tapes, and then hook them up to the network.

https://www.quantum.com/en/products/tape-storage/

From this perspective, you buy as many tapes as you want storage (aiming for 500TB? Buy like 40 LTO7 tapes for your library. Aiming for 1000TB? Buy 80 LTO7 tapes for your library, assuming compression of course).

From there, you just read/write to the library, and have the underlying software handle the details, like any other NAS.

> assuming compression of course

I don't know why this seems the standard practice in the industry, but it really annoyed me when I realized a “15TB” LTO-7 tape has actually only 6TB real, “native” storage coz it assumes some average compression ratio.

Why is this acceptable? What if I use the tape to store incompressible data like video and images? Feels like intentional cheating.

Meh. It's only cheating if the manufacturer keeps it a secret, and they don't.

When a company is spending >$20k on a tape system, the people in charge of buying it will talk to the sales people, tell them the use case, and get a more accurate estimate.

If the buyers are engineers that understand the concept of storage space measured in bytes, this 'estimate' does not help them. If they don't, it only serves to mislead.

The fact that buyers may talk to salespeople is really not an excuse for the deceptive behavior.

"well, in our restaurant medium rare means well done and rare means medium rare, but that's okay our customers are well-paid professionals, they'll talk to waiters to get a more accurate picture"

Tangent: That's how most restaurants operate, because most people don't know what the words mean and get upset when they ask for "rare" and get what they asked for.

Clothing manufactures also lie about the waist measurements of pants. (Go measure yours and see.)

Funnily enough the first hard drives were like this too. They were a round pizza-sized (but a few inches thick) replaceable cartridge and could be inserted into a 'reader' which contained the heads.

But I guess work the data density required these days and the extreme closeness of the head to the platters this won't work anymore as dust would get in.

But this is how the first hard drives worked, we had some with our pdp-11 at the computer museum.

Tape has an inherent density advantage over hard drives because tape stores data in the volume - tape is a 3D storage technology. Hard drives are limited to recording on a handful of slices through a cylinder inside them; tapes record on the peeled surface of a cylinder roughly the same size. One tape cartridge contains more than 30 times as much surface area as the highest-density hard drives for recording data.

The trade off to all that capacity is latency. But for archival purposes, it doesn’t matter if it takes 2 seconds or 2 minutes. As long as the writes are quick enough to keep up with the data.

And tape has (historically at least; I haven't check in for a few years) been faster at purely streaming loads than spinning rust. Pre SSDs, there'd be systems that streamed to tape to keep up with input data, then would offline load that into spinning rust databases to crank numbers where seeking dominated.

Yup. A well-known issue with tape is that machines have a hard time supplying it with data fast enough to keep up with its linear travel speed when creating the backups.

I believe that LTO can actually regulate the speed of the tape

It can, within reason. Also source machines (and more importantly, source media) are much faster in the first place this side of 2010.

Yup. I did some tests and my SSD (and even HDD) can handily keep up with the data rate. If not, I can always add a small staging drive the size of a tape to pre-prepare the data for writing.

> Gets me thinking tho: Why stop at 5.25in drives? Let's go back to washing machine sized units or even bigger. Let's stack platters so dense and heavy that we can use the spindle as a flywheel UPS system too.

Because modern 20TB hard drives already take like a full week to read from beginning to end (aka: time for a RAID6 rebuild), which is too long as it is.

The problem with hard drives is that they need "more read heads per byte" than the current technology. You can solve this by making more hard drives (ex: RAIDing together smaller drives, say 4x5TBs), so you have more read/write heads going over your data faster.

There's multi-actuator hard drives coming up (2 independent heads per drive), which will report themselves to the OS as basically 2-hard drives in one case. I think that's where the storage market needs to go.

------------

Tape is the king of density: having the most bytes of storage on the fewest number of read/write heads possible. But Hard Drives encroach upon that space, especially with like 20TBs on one read/write head.

> Or how about making a truly gigantic tape spool? Perhaps kevlar threads impregnated with iron could serve as "tape" and we could spin kilometers of that onto a spool.

There's no advantage to that. A stack of 300 LTO-tapes will practically use up the same amount of space as one-tape that's 300x longer. (Besides, LTO-8 tapes are 960 meters long, roughly 1km. The idea of pulling on a thing that's 300km long and hoping for it to not rip itself apart is... pretty crazy. There's definitely some physical constraints / practicality with regards to just material science: shear/stress kind of calculations)

The "jukebox" concept... really a tape-libraries (robot that picks tapes out of storage compartments and shoves them into the drive) is the ultimate solution to density.

I never understood why they didn't make a hard disc that operated its heads in parallel and distributed data accordingly. Still one assembly, but using all the resources it has.

You'd have (for example) a four-platter, eight head drive, and instead of storing a given byte serially at cylinder 123, head 0, sector 5, bytes 0-7, you'd store it at cylinder 123, sector 5, byte 0, and light up heads 0-7 all at once.

Now, this might have been hard with early logical drive designs that tracked the physical geometry and expected a more serial format, but that was long ago hidden beneath LBA translation.

Maybe there's a power or crosstalk issue with activating 8 heads at once.

because at those scales heads on the same arm move in relation to one another due to vibration, thermal expansion etc

I have a need to store a few hundred TB of data, and tape is extremely expensive for this, since it has a $5k+ upfront cost.

So I just buy external $200 USB hard drives.

There is a market for something lower latency, higher capacity, but pay as you go (no upfront cost).

Frankly I wouldn't mind some sort of giant disk drive, vinyl record size, if that makes sense and is lower cost/TB.

Yes. There was a time when the tape drive wasn't so expensive; the densities weren't there either but that market climbed the wrong side of the curve in a couple of ways.

The modern tape drives are miracles of precision and the tapes are these cool little cartridges but why? The meat and machines handling tape carts liked the of 3/4in format just fine. With modern electronics for the write heads and slightly less ruinously expensive loading and running hardware we could still get decent data density, even if we have to hunt for the tracks each tape load.

It didn't used to be this way. Tape was once affordable for individuals; ca. 2000 Travan was marketed to consumers and DDS wasn't too much of a stretch for prosumers. But they're all gone now and only LTO is still standing. Given the small, cornered market segment that tape now serves, there is no incentive to bring the price down. I'd like the option of being able to manage my own backups locally instead of using cloud storage, but the economics of Glacier remain too good until the storage size is several times what I could foresee needing.

My tape setup cost less than a single external hard drive. Older tape techs come down in price in time.

My company has built something for that.

Basically you have a nextcloud instance that you fill up with your data, then you ask for an LTO tape to be written with it. Repeat ad infinitum.

The tapes are yours; when you buy a tape it comes with 3 free operations (read, write, checksum control). If you want your tapes back, we'll mail them to you, else we keep them in storage for you.

As the tapes are written using LTFS, you can easily read them back anywhere with the proper drive.

You only pay a fee for the cloud storage; it also gives you access to your archive's database (what file is on which tape, etc).

That sounds fascinating. Where can I learn more? Other than writing the same data out to multiple tapes, do you have any capacity for redundancy, should a tape go bad?

Yes you can ask to have several copies, as you simply pay for the tapes. The archival solution can store data also on disks or any S3-compatible storage.

We also implemented the ability to turn off entirely (cut the power) disk chassis when unused, to save power (using 60 or 102 drives chassis). So if you have enough data (a few hundred terabytes to petabytes) it can make sense too.

Is it https://www.intellique.com?

You can find the archival solution at intellique.org and open a free account on demo.intellique.org . The whole solution is OSS, for obvious durability reasons.

It’s true tapes are okay when time is not of the essence, but there’s currently a lot of pain dealing with tapes when it comes to loading reading writing and verifying everything. You need some way to make sure the data on the tape matches the original after writing. Then you need some way to verify the copy you get back from the tape matches. You need to have multiple copies because sometimes whole tapes go bad. You also have a limited number of times you can try to read or write the tape without too much wear so you have to have a plan for how to read or write those things efficiently. This is stuff that can be mostly automated but it’s surprising to me that mostly this is done manually and the tools that do exist are often bespoke in one way or another. A nice thing about a hard drive is that most of that logic is baked into the firmware of the drive so you are less dependent on having a good system for verification and record keeping (though you still need it). Tapes can currently take minutes to load and hours to read or write if you need to copy the whole thing. I’m sure if someone was going to make those dice of drives they’d figure out the economics and logistics but it would be a monumental effort both technically and physically to make it work well.

> You need some way to make sure the data on the tape matches the original after writing.

Do you? Virtually every filesystem has some kind of error detection, maybe even error-correction built in.

That seems like a solved problem to me. CD-R solved this by writing Reed-Solomon codes every couple of bytes so that if any error occurred, you could just fix them on the fly. (As such: you could have scratches erase all sorts of data, but still read the data back just fine)

I have to imagine that tapes have a similar kind of error-correction going on, using whatever is popular these days (LDPC?). Once you have error correction and error detection, you just read/write as usual.

-------

If Tapes don't have that sort of correction/detection built in, you can build it out at the software level (like Backblaze used to do)... or maybe like Parchive (https://en.wikipedia.org/wiki/Parchive).

> or maybe like Parchive

par2 even has options for specifying level of redundancy. I've had good experience in recovering large corrupted files from an external drive - since then, I've incorporated it into the automated backups of my personal infrastructure.

https://github.com/Parchive/par2cmdline

> Virtually every filesystem has some kind of error detection, maybe even error-correction built in.

Nitpick: you mean physical encoding, not filesystem.

Not nitpick: it’s nowhere near enough. Blu-Ray bitrot is a huge issue and if you don’t either write out your data twice (to the same disc or to distinct discs) or use PAR2 or similar, your backup isn’t worth the money you paid for those shiny coasters.

There's special archive Blu-Rays (M-Disc I believe) which are meant to last 100 years.

Not sure if they really do. We won't know for another 90 years or so :)

Yeah I’ve written about those on here before. I bought some but haven’t had time to use them; I’ve read anecdotal reports of them not lasting any longer than regular HTL (instead of LTH) discs.

Tape, at least right now, seems to have a pretty steep entry barrier though. Tape drives cost thousands of dollars and each tape itself seems to be fairly expensive too. About $50 for an LTO-7 tape that's 6 TB. That's $8-9 per TB just for the tape. HDDs seem to be around $20-25 per TB.

Need to 300 TB of capacity?

50 tapes at $50 totals $2,500 + a few thousand dollars for the tape drive.

Or you could buy about 12x 18TB external drives for about $450 each totaling $5,400. About the same.

For home use, you can probably get an LTO4 setup going for less than $250 (including some used tapes). LTO-5 will remain write compatible with your tapes and LTO6 will remain read compatible, so there's a forward migration path that will keep things cheaper.

Quantum's marketing for their BigFoot 5.25" drives was interesting.

They argued that a 3600 RPM drive was as fast as 5400 RPM 3.5" drive because you're spinning a larger diameter platter. Technically true.

They also claimed that since you could fit more data on a track, since it was larger, you would do less seeking, and that's what makes mechanical hard drives slow.

This didn't stop BigFoot drives from being slower than 3.5" drives that were 3 years older and 1/4 of the capacity.

I loved my bigfoot drives. Yes they were slow, but they was cheap and ran cool and (excepting certain lines) proved amazingly durable. As in, "ran as the mail spool disk for a decade" for a system sending ~5k emails a day. That one i think i still have on a shelf here somewhere with a "well done thou faithful servant" postit note

In my time doing tech support and repair, they were probably the most replaced drive to failure that I ever experienced. At least until the Deathstars came out.

I think i was buying refurbs then. as i recall the issue was with the controller boards. i know a bunch got used as cheap system/boot disks on alphas at CGI render farms, too: some of my stock came from surplus "Titanic" render nodes. I recall they had special cases for them; 24 pack pink foam crates. I had at least two of the crates...

Compaq must have been a big buyer of them at the time - they used them in quite a few Presario models (home models, cost cutting I'm sure)

Yes I remember those foam crates. 10% of a new crate of Bigfoots were DOA right out of the box. But I guess the ones that lasted more than a week were much more likely to last years.

> Why stop at 5.25in drives? Let's go back to washing machine sized units or even bigger. Let's stack platters so dense and heavy that we can use the spindle as a flywheel UPS system too.

Current HDD tracks are around 50-60nm wide, and their limitation is not magnetic grain density (those are around 10-12nm for non-HAMR/MAMR substrates and even smaller for HAMR/MAMR substrates.) The limitation is the long (relatively speaking) actuator arm trying to precisely stay on a very narrow data track. There's actuator flexure, disk flutter, and aerodynamic noise, all of which increase with increasing platter radius. This increases your minimum track width, and minimum disk thickness/minimum disk spacing and end up defeating the purpose of bigger disks in the first place.

Also it's important to keep in mind that in order to manufacture a drive, the entire drive is written to and read back multiple times, which takes an increasingly long time, creating a huge push for some steps to be combined.

That all being said, I'm not entirely sure where the sweet spot in terms of data density per unit volume. It might be slightly more efficient with 5.25 drives. But I can assure you that the record sized or washing machine sized HDDs of olden days are not practical anymore ...though they would be dope if they did exist!

Tape is great for large-scale setups, with the cost all being in the drives and media being dirt cheap. But what if I just want a backup of my personal or small business's data to put into a safe. Even regular hard drives are far more economical than spending a couple thousand on a previous-generation LTO drive, or many hundred on an even older model.

I was going through that same dilemma but they’re really not perfectly comparable options. Drives are more sensitive to storage and handling damage than a tape cartridge is.

An LTO4 drive is capable of writing 800GB raw capacity cartridges, and it can be had for ~$120.

LTO4 cartridges go for about $10-20 each. At about $15 they break even with hard drives in price per GB. If you were willing to go with used hard drives the break even point is closer to $10 per cartridge.

It's useful if you get a good deal or if you like the unique advantages of tape (smaller medium with decent shelf life), but otherwise the price advantage seems dubious.

I'm curious as to whether there would be a market for a new, more modern form of tape-based storage, one that incorporated ideas from modern HDD design, had a lower barrier to entry, and/or was based on open-source hardware outside of an existing proprietary ecosystem.

Reply on news.ycombinator.com