Out of Stock: How to Survive the LTO-8 Tape Shortage | Hacker News Reader

Out of Stock: How to Survive the LTO-8 Tape Shortage

backblaze.com

101 pointsingve74 comments

At this point you are probably buying an IBM LTO tape drive no matter the label on the front. If you're buying a library, I don't really see any reason not to go with IBM "Enterprise Tape". It blows LTO out of the water in capacity and performance, and the media upward and downward compatibility are astonishing https://en.wikipedia.org/wiki/IBM_3592

LTO was an interesting period and probably most useful for media professionals that have single tape drive decks. But the real market for tape is massive silos where the tapes sit reliably, dutifully, and not using any power while holding massive quantities of data. And if you have to use a "proprietary" tape cartridge and drive to do it, I fail to see how that's any different than the proprietary disks, disk bus interfaces, or flash and FTLs. On the other side you're talking FC or SAS or Ethernet, same as disk/flash talking SAS or NVMe.

I'd use Backblaze a LOT more... if consumer ISPs in the US weren't pretty much monopoly BS. (Can you really call "up to" 12mbit down (and like 1mbit up max) VDSL a competitor to 1000mbit down ~30mbit up cable? Both of which are WAY TOO SLOW for backup up real workloads! Let alone the caps @.@)...

Also no, where I live is typical of American Surburbia, if anything it's actually slightly better than the median who suffer through Ex-GTE hell and cable ISPs so bad they make even comcrud look good.

> Both of which are WAY TOO SLOW for backup up real workloads!

If you use half of a 30mbit upload for backing up, that's 5 terabytes a month. I think that covers a lot of real workloads.

Except that if you saturate your uplink you will also likely make your downlink sluggish.

And a user may not want to wait 20 hours for an operation to complete.

If maxing your upload slows down your download, or vice versa, its an issue with your router, anything sensible should put maximum priority on TCP acks and avoid the issue altogether.

ACK prioritization isn't always good for overall usability of an internet connection. It definitely shouldn't be used unless there's good FQ-style QoS active in both directions. Once you have that, if you're going to do anything special for ACKs, you should probably try ACK thinning before prioritizing ACKs over regular data transfers going in the same direction.

what is an example of when it is bad to do this.

If you have enough ACKs going upstream to start being a problem, you are most likely downloading a large file, and that traffic isn't particularly latency-sensitive in either direction. If you have something actually interactive that's also trying to use the same link (eg. VoIP), that traffic probably should be prioritized over a firehose of ACKs even if it will make the file download a bit slower. ACKs shouldn't be treated as any more important than the payload data they they are acknowledging. Low priority downloads should generate low-priority ACKs.

ACKs are generally so small its irrelevant, unless your bandwidth is ridiculously asymmetric (less than a 23:1 ratio 64B ACK vs 1500B packet) which even really crappy cable providers don't do.

It's a very real problem for many if not most surviving DSL connections. Cable has gotten a bit better in recent years, but 10:1 is still about the best you can normally expect, and 20:1 or 25:1 is still common for some of the higher speed tiers on cable. So ACKs can realistically be expected to take up half of your upstream bandwidth on common connection types. That's far from irrelevant unless you never transmit anything other than ACKs and HTTP GETs.

To be fair, the example we're responding to was that asymmetric.

Also for reference, most TCP stacks default to sending one ACK per two packets received.

But ACK aren’t the only upstream traffic. Latency gets added to all your requests.

What are you trying to say? What latency is being added to what requests? Are you arguing for or against ACK prioritization?

I am neither arguing for or against, just that delayed ACK probably isn't the only reason for the connection to become sluggish. A typical page makes dozens of requests, these are packets to be uploaded, if the uplink is saturated, the latency should increase, I'd expect that to affect the time it takes to render the page.

Backblaze has a bandwidth limit setting if this is a concern.

Yes - your first upload is going to be painful, but for the average user the following incremental backups are just a blip.

And if you're a business and have lots of files to backup, well, it's leased line time.

Took me the entire 15 day trial period for my initial backup of ~4TB. After that though, I never notice it uploading the incremental backups.

> Except that if you saturate your uplink you will also likely make your downlink sluggish.

That's why I said "half". And if you make 1TB per month then it's only 10%. You could restrict it to overnight too.

> And a user may not want to wait 20 hours for an operation to complete.

Maybe. Tape backups are often on a daily basis anyway.

Is this normal for Windows? I assume you are talking about Windows. Maybe it depends on the quality of your network stack?

I've been maxing out my uplink 24/7 for the last 10 years. My upload speed has increased from 15 KB/s to 100 Mb/s in that time, and for the life of me I could not feel or measure any difference in downlink speed or perceived performance after switching to about 1 Mb/s and up.

and after 1TB your ISP will charge you extra or claim you're "probably doing shady things" and disconnect you.

Prior to Backblaze, I used another service that shipped me a hard drive for the initial seed. 1 TB is painfully slow on my 100/5 connection.

Unfortunately they stopped all home sales, so it took weeks to seed to Backblaze.

Google and AT&T are taking their sweet-assed time laying fiber in my area, if they ever finish at all. 1G/1G is really the only solution.

If I'm not mistaken, Google has called off any plans for expanding Google Fiber.

https://arstechnica.com/information-technology/2016/10/googl...

They claim to have not stopped in KC, but in my area, the city has lengthened right-of-way permits out to 2025. Those were typically only issued for 1 year.

http://overlandpark.maps.arcgis.com/apps/Viewer/index.html?a...

> ~30mbit up cable

If you think this is bad, I recommend you stay 100' away from Canada at all times.

Some of us are lucky enough to live in cities with the foresight to install municipal fiber networks. I have 1Gbps symmetrical at home, which makes backup to the cloud very feasible.

A lot of people live in those cities (in my case, the publicly owned electric co). Then they sold it all off to Rogers. Anyone want 100/100 for $800/month?

Perhaps Elon's Starlink will save you at some point. (being serious here, the network should be able to support much better than cable.

Sort of going off on a tangent to Backblaze's post. I know big enterprises still use a lot of LTO tape. But the total market worldwide for LTO drives and LTO tapes (two manufacturers, total!) can't be very large compared to the number of spinning rust type hard drives that are manufactured yearly, and sold into distribution channels. Economy of scale matters a lot. If you can buy a product that's manufactured in the dozens of millions and piggyback on everyone else's purchases, it may be a better solution.

Backblaze's system architecture is fairly failure tolerant, so they use a huge number of "consumer" grade drives that would cause enterprise people to run away screaming. They're successful with it.

I know a number of people who have self built ZFS servers in the $2000 to $18,000 price range, who also use consumer grade drives with a failure rate no higher than the expensive $450 enterprise drives.

A small video production company I worked for used LTO tape for archiving footage, so it's not just big enterprise.

Small sample size, but I consult on backend systems for post-production houses / special effects contractors in the Vancouver BC film industry, and tape is exceedingly rare.

Also, there is only one manufacturer of LTO tape drives. IBM. The other companies ship OEM versions of the IBM drive.

I have several ZFS Servers in my Homelab, using both consumers and gasp referb drives....

I do not have a higher drive failure rate in my homelab than the enterprise gear I manage daily

at the end of the day the mechanical parts between the drives are very similar, the biggest difference if the firmware loads on the different types of drives.

Years ago, a vendor revealed that the largest buyer of tapes on the planet was Google, not some enterprise.

I think where tape starts to have an edge is in long term storage, particularly for archival. You have to take into account that a tape sitting somewhere in a controlled environment will be readable for quite a long time compared to a hard disk.

> Backblaze's system architecture is fairly failure tolerant, so they use a huge number of "consumer" grade drives that would cause enterprise people to run away screaming.

I think what would "cause enterprise people to run away screaming" is that your data is stored by them in only one data center, and they don't offer a S3 compatible API (although someone may have addressed the latter).

Backblaze however correctly notes that if you really want redundancy, you get that from using two or more vendors.

Tape is physically a lot more robust than hard drives, the single endurance advantage the latter have is better tolerance of environmental extremes, heat and humidity.

BackBlaze is absolutely genius at marketing. They are one of the few companies that know how to produce “ads” that are both super interesting and useful while explaining how their service can solve a real need.

I absolutely love their hard disk stats as well. Whenever I read one of their articles, I get the warm fuzzy feeling that these people really know their stuff and that my data would be safe with them.

Kudos for building great service and marketing in such a useful and informative manner.

It's really a textbook example of content marketing. Mind you, they have a good service to go with it. But the visibility they get from things like their hard drive stats is worth a huge amount.

It is also of immense worth to anyone buying drives.

Those kinds of stats are impossible to get by for mere mortals so we typically only have anectodes to go by.

So for that the visibility can not possibly be too much nor undeserved.

People severely underrate content marketing in my experience. Or misunderstand it. Just having a popular technical article on your business domain is good. You don’t have to do anything clever in the article itself marketing your business.

I miss Basho: we’d frequently post quality distributed systems content on our blog, and running the RICON distributed systems conference was quite a privilege.

Backblaze and Lensrentals are shining beacons of content marketing.

Cloudflare pretty good at it also.

It's content marketing as a positive rather than a negative force.

Good content marketing should be a positive force. Done well, it's win-win. Readers (including potential/current customers) get interesting/useful content; you get your brand out there at usually fairly low cost.

> It's really a textbook example of content marketing

It's a textbook example of excellent content marketing. Not many services can do it like Backblaze, because not many services have something this interesting to write about.

I've always been impressed by their articles describing how they build their storage pods. https://www.backblaze.com/b2/storage-pod.html

> By March 2019, US import bans of LTO products of both manufacturers were in place.

Never underestimate the profit of a suitcase full of tapes hurtling through the airways?

Are you suggesting smuggling in LTO-8 tapes?

No, but I know a guy, and he can meet you in the alleyway

The alleyway in the ocean?

AlleyExpress.

That was a reference to the famous quote, "don't underestimate the bandwidth of a station wagon full of tapes"

Do any cloud backups provide sneakernet uploads by mailing drives to them and they return them?

I think Google and Amazon both offer that for their storage products. I imagine most large providers have something along those lines.

Edit 4: And because it wouldn't be right to mention all those and not Backblaze's own offering in a submission spurred by Backblaze, https://www.backblaze.com/b2/solutions/datatransfer/fireball...

Edit: AWS: https://aws.amazon.com/snowball/disk/details/

Edit 2: GCE https://cloudplatform.googleblog.com/2018/06/Bust-a-move-wit...

Edit 3: Azure https://azure.microsoft.com/en-us/services/databox/data/

Yes: https://aws.amazon.com/snowball/disk/details/

You can go much, much bigger, like a petabyte at a time... https://aws.amazon.com/snowmobile/

Most do, Prices and service vary.

Backblaze does as well for their B2 service

https://www.backblaze.com/b2/solutions/datatransfer/fireball...

This is the kind of thing that really makes me want to have somebody please clone and sell them out of a country that doesn't really care about patent infringement. They can at least supply tapes until the big guys sort out their petty battle. They'll both be sorry for suing each other after this third party takes all their revenue for the time being.

Users shouldn't be the ones getting punished for this.

> have somebody please clone and sell them out of a country that doesn't really care about patent infringement

How would one “sell out of a country” when it’s an import ban?

“By March 2019, US import bans of LTO products of both manufacturers were in place.”

It’s not obvious from that sentence whether the ban is on LTO-8 in general, or just from Sony and Fuji. It’s quite possible it’s only the latter.

From previous reading, both are banned, Fuji recently: https://www.theregister.co.uk/2019/05/31/lto_patent_case_hit... (Although previously I remember it was Sony that declined to start up LTO-8 production, after getting their importation of similar BaFe LTO-7 tapes banned, maybe el Reg got which was which confused in the article.)

I don't know much about this topic, but if it was so easy to manufacture the cheap LTO tapes, there would be more than two manufacturers left in the industry and there definitely were non-Japanese companies.

Who's the biggest/most security conscious company that does cloud backup at machines not controlled by themselves?

i.e. it would very interesting to me, and a huge endorsement of the process itself, if Google did some cloud back up on Azure or vice versa.

Tldr: it's about patent related US import ban of tapes and voluntary vendor initiated US import stop of drives, not general unavailability.

gr8. Sony and Fuji have a fight and they'll both lose since people committed to LTO tape will move on. Gotta love those patent lawyers.

What will they move to?

The cloud (company who wrote this article) or hard drives.

Can hard drives store long term like LTO tapes?

The article (which is obviously biased keep in mind) makes the argument that even LTO isn't really long term due to compatibility issues between tape generations - you have to keep migrating your tapes since new machines can't read older tape densities.

Realistically, companies keep making older generation tape drives for a long time, for example, HP LTO-5 drives are still widely available from normal vendors like Newegg. And except for one discontinuity that's pretty clearly due to the change from metal particulate to BaFe, which is at the heart of this patent dispute, LTO offers two generation back read capability, and one generation back write capability: https://en.wikipedia.org/wiki/Linear_Tape-Open#Compatibility

Backblaze use hard drives, and there system seems to be incredibly fault tolerant and long lasting.

I have read external hard drives that sat unused for 15 years. Hypothetically the bearings might dry out or something but I haven't seen one fail.

One the other hand I have chronically had problems with tapes I couldn't read, had spots of data corruption, or that took 18 hours to restore a 1500 byte file.

Part of it is that the use of tapes is less routine so it is a common mistake to think you wrote a good tape but you didn't check it and it's unreadable.

Another competitor for LTO tapes is BD-XL, which is more expensive per unit TB but the drives are cheaper.

What generation of tape technology exhibited the problems you describe? I've never suffered such problems with original 1/2 mag tapes, DDS, or LTO.

BD-XL is wildly more expensive than LTO, although that won't matter if you can't get LTO to be reliable for you. Just checking now, B & H Photo which tends to have good prices and a good supply chain, $54.57 for 10 100GB BD-XL or a terabyte (https://www.bhphotovideo.com/c/product/1101544-REG/verbatim_...), not so cutting edge and still with competition LTO-6 from Fujifilm, quantity 20, $11/TB (https://www.bhphotovideo.com/c/product/1096090-REG/fujifilm_...). We also have a problem with no one worth trusting making cheapest single level BDR discs, the least worst is CMC, and I wonder how long before BD-XL sufferer the same problem.

Did you not read TFA? It's explicitly an ad pitching cloud storage as a solution to customers fed up on waiting for Sony and Fuji to sort out the lawsuits surrounding LTO-8.

Does this reduce the prices of lto8 drives?

Reply on news.ycombinator.com