Hacker News Reader

Comment by vlovich123 | Hacker News Reader

What’s the point of reporting compressed capacity? Isn’t data typically compressed at rest, negating any value of the disk doing it?

View on HN

After having helped on a lot of tape storage implementations, I get why it's useful for quick estimates on backup solutions.

Compression really is a game changer when you're over a certain amount of data you need to protect, and backup technology does some really clever space savings at basically all stages, making it hard to find a good and easy metric to help sell to management. It's a lot easier to tell management:

"These tapes hold 50 TB of production data, and we have about 45 TB in production. Backups and restores will be faster, we'll have fewer tapes to manage for backups"

than it is to explain the different types of data you need to move to tape, the expected compression ratio, etc.

That being said, I do agree they market it too much and it gets confusing, as it should only be a reference for implementation and quick calculations.

Surely how well it compresses depends on the material you're backing up

It seems they went all in on the "compressed" value after missing the doubling of the real space in LTO6.

Especially if you backup encrypted data (which isn't unusual - to have data encrypted at rest) you will have a compression ratio of 100% or even more (metadata of the compression needs to be stored too)

The drives will do encryption too so that you don't have to pick between encryption at rest and compression.

From the SSL world we know that encrypting compressed data can lead to vulnerabilities in the encryption. Does that nog apply here? https://en.wikipedia.org/wiki/CRIME?wprov=sfti1

It doesn't really apply as both CRIME and BREACH rely on small N on the payload size, and heavy use of repeated chosen plaintexts to iterate through the secret state space.

Isn't it harder when you have to later report to management that actually, your 45TB of production data do not fit on the 50TB tape ?

I suppose compression ratios vary a lot between users of these things, right ? And if you are backing up user-generated data, you don't know in advance how well it will compress.

In this case "The latest TS1170 drive supports 50TB JF media cartridges with 150TB compressed capacity through 3:1 compression"

So looks like 50TB raw if I am reading this right.

For these sorts of solutions you are usually not using just 1 tape. The tape is the cheap part. The drives cost a decent amount though. So in your case if the data does not fit. You buy another tape and pop it into the existing drive and finish the backup. Also you would have some sort of rotation schedule and at least a few dozen tapes. If you are doing it right offsite and onsite rotation too. An extra tape in the mix of that is not something they would worry about. When I did this for a small org I had the weekends to do a full backup of all of our computers. Then during the week I would do incrementals. At the end of the week I would ship off the prev week to an offsite and start the next full backup. Not perfect but gave us a onsite 1 day return offsite (building gone) 1 week out. The offsite would keep about 6 months of tapes and return them about once a month.

Yeah. For most operations, you pay for drive throughput & redundancy, and the actual storage itself is “too cheap to meter”.

Except when it isn’t, but those customers know who they are.

These are the same customers that only make 1 tape of the data. One facility I was at required 3 copies written by 3 different writers. These were in large robot operated units, so it was all automated by our front end. One copy stayed locally, the other 2 went to 2 different offsite locations in different parts of the country with the closest being in another town across state. Each tape had to be removed from the writer and placed into a different drive to do a read back for comparison. In the LTO4 days, we would catch lots of write errors like this. I wonder if the newer formats are any better about these errors? How I miss the old days of tar/mt commands, er, no, who am I kidding!

In the previous century we'd restore the production tape backups to a "production playground" that verified the backup actually worked as well as give a production-like testing environment.

Odds are you already have a backup system and so you know from experience with your old system how much compression you really get.

>After having helped on a lot of tape storage implementations, I get why it's useful for quick estimates on backup solutions.

Can't you just do that estimate in your head? If you know your data mostly text and has a 3:1 compression ratio, when you buy your tapes you can multiply the raw capacity by 3, if it's mostly compressed images (i.e. gif, jpg, etc), you're probably not going see much compression at all.

Does going with some tape drive manufacturer's estimate really help you?

> "These tapes hold 50 TB of production data, and we have about 45 TB in production. Backups and restores will be faster, we'll have fewer tapes to manage for backups"

Sure, if "production" means database. Which is sometimes sensible and sometimes extremely far off.

> than it is to explain the different types of data you need to move to tape, the expected compression ratio, etc.

To some extent, but you mostly just need two numbers. Database capacity and photo/video/audio capacity.

Not once in my days of using tape storage have I ever enabled compression, so it always seems like a waste to my specific use case of storing large files of media formats, specifically video. Unless the video was in some sort of RAW uncompressed format, which is close to never, and video in a CoDec would not benefit from this compression.

Tape compression is probably not meant for your use cases, it's likely meant for large government institutions or corporations that generate petabytes of uncompressed text data a week - so instead of needing to manage compression as a separate process, the tape drive just does it without any other processing needed. The data can go from collection/creation to long-term storage without any other steps in between. For some organizations that's a big win. For the average home user backing up their plex, it might seem unnecessary.

Lots of home users backing up their Plex with an LTO5 or greater are there?

I am one of those home users with an LTO5, but I do not use tape compression and I have no illusions about what it was made for. I write tapes without any compression, because the stuff I write can't be compressed. Tape compression wasn't made for my use case, it was made for organizations that generate petabytes of text data.

There's still a fair bit of mainframe-format data (zoned decimal, etc.) out there. These are typically in defined-data formats, a/k/a hierarchical , which is sort of a hybrid between data files and text files ("records" are lines, but often with multiple formats within a single file). These can achieve compression ratios ~= 90%.

Standard written text typically runs about 50%.

Compressed audio, image, and video data of course compress very little, if any.

   tank                    compressratio  1.14x  -
   tank/backup             compressratio  1.43x  -
   tank/movies             compressratio  1.01x  -
   tank/music              compressratio  1.01x  -

As you can see, my actual file backups (servers and clients) compressed somewhat, but movies and music I got nothing (this is from ZFS but tape compression will be similar).

Back when I still managed an LTO tape library (which was way back when LTO-4 was state of the art), our data was encrypted before writing it, also negating any compression. And back then the compression ratio was advertised as 2:1, not 3:1.

This was before the introduction of the much lauded "Middle Out" compression.

Purely legacy, that's how it's always been reported.

It takes advantage of shallow purchasing decisions.

If all your competitors report 30 TB*, and you report 15 TB, you may have a better product, but what the people with purchasing power want is bigger numbers, and you advertise a lower number

*with 3:1 compression

If so, why not state 100:1 compression on the canonical FF*N data set.

Because contrary to popular belief, people are only finitely foolish

But in fact this did happen to some extent. LTO tapes used to advertise with 2:1 compression ratio. Then that became 2.5:1. And now IBM is doing 3:1

And that's not because we just discovered how to improve on Lempel-Ziv. The switch to 2.5:1 was around 2012. LZMA is 20th century tech, and people still use gzip today. Compression technology doesn't account for the advertising number inflation there.

Because all the companies have to do it in lockstep (see "reach vs standout" recently in tape measures).

Because nobody has yet bothered to sue them for false advertising, so they continue to get away with it.

Bigger number for marketing

Reply on news.ycombinator.com