How to get your backup to half of its size – ZSTD support | Hacker News Reader

How to get your backup to half of its size – ZSTD support

percona.com

Today we are glad to introduce support for a new compression algorithm in Percona XtraBackup 8.0.30 – Zstandard (ZSTD).

Results shows that ZSTD not only overcame LZ4 results on all tests, but it also brought backup size to half of its original size.

When streaming is added to the mix is when we see the biggest difference between both algorithms, with ZSTD overcoming LZ4 with an even bigger margin.

This can bring users and organizations a huge amount of savings in backup storage, either on-premises or especially in the cloud – where we are charged for each GB of storage we use.

Nice! I've always had a lot of respect for the Percona team. If you haven't checked out the product I suggest you do :)

We did a similar implementation of Zstd on ScyllaDB some years back. You can see how it compared to LZ4, Snappy and Deflate.

The interesting thing in comparing compression algorithms is — "what are you optimizing for?" Compression speed? Decompression speed? Storage savings? There are Tradeoffs. Getting the most storage savings might be suboptimal with a database if it adds too much latency or hampers throughput.

Nicely done, Percona!

https://www.scylladb.com/2019/10/07/compression-in-scylla-pa...

You may get a quite different answer in 2022 as there has been significant work on the raw performance of the zstd code. On x86 with BMI2 it is noticeably faster.

Good point!

My main pc is a MacBook 2013 and brtfs with zstd has changed the way I use it drastically. It basically doubles its storage for common day things. Sure movies and audio and such don’t compress but everything else does. Makes it feel a tad snappier as well.

I guess you running linux on it ? Or it is fuse shenanigans?

Arch Linux. Linux has support built in and so does grub.

What else do you store besides movies and audio so that it becomes noticeble for you? I also use btrfs with zstd, but mainly see the difference for OS/apps files. Movies and audio don't compress as you noted, and code and scripts take tiny space anyway.

The hard drive is 64 gb. I have 20 left over. The other 40 is the base arch os install and all my apps with the files I’m working on.

What's surprising to me is that zstd beat out lz4 on compress/decompress speed. LZ4 is supposed to be purpose built to optimize on those metrics. Great job zstd developers to get perf that good!

The lz4 developers are the zstd developers :) In this case I suspect the IO reductions on the output are what matter -- while lz4 is faster, it also produces more output (of course) which you then have to write to disk. This can make the wall time take longer... basically you become IO bound on output, be it network or disk.

zstd also can tune itself and adjust ratios to saturate output bandwidth, which is pretty cool (--adapt).

That is exactly it. The difference comes from compression ratio. ZSTD is compressing the data more, so you need to write less back to disk. Also when talking about streaming, the difference is even more as it has to go over wan to S3 (provider used on blog test).

So we've got a system that backs up to an XML file nightly. I noticed a couple weeks ago the bucket was using a not-insignificant amount of disk space.

These XML files are full backups rather than deltas - so each one contains the full previous file plus the new additional data.

My assumption was if I grabbed just a swath of the old ones, say a years worth and compressed them together I would get really decent savings.

I was pretty disappointed - gzip knocked 10 gb off of about 100 gb of data. I started doing some research and found people saying 7zip and it's sliding dictionary size options were the answer. After multiple tries, each run of 7zip taking multiple days I was able to get it down to about 70 gigabytes from 100. Better then gzip but frankly nowhere near what I would expect.

Does there exist a compression that could better handle this sort of expanding documents?

That doesn’t make much sense, window size or not. Even base64 encoded random data would be about that bad.

Is the XML wrapping a bunch of other random data or something?

base64 random data would be that bad only because of base64. Random data does not compress at all on average.

As an example, I have zstd enabled on some zfs pool. The client-side encrypted time machine backups does not even compress 1 %, as expected.

Yes, that’s why I said that?

You can’t embed binary data in xml raw, so a common pattern is embedding it as base64, or similar type of wrapping.

Technically, it’s possible to escape it in other ways, but it’s error prone.

Either way, XML which has text strings or whatever typical XML document data that ISN’T something like base64 encoded random data should compress dramatically better than what the poster was talking about.

So, what the hell is in your XML anyway?

Ah sorry, I misunderstood what you said

Thanks for responding. Apologies for the cursing to add excessive flair.

You need what's called "solid compression", meaning that the files are concatenated together before compression. Because otherwise compression restarts from zero for each new file - https://en.wikipedia.org/wiki/Solid_compression

Some compressors have options for solid compression, or you can use tar to first concatenate the files. For both, you need to sort the files first, so that they are compressed in chronological order, otherwise the common information will drop out of the dictionary.

And how big is one file and how much RAM you have? Because you might need increasing the "dictionary size" of the compression algorithm, both 7zip and zstd support multi-gigabyte sizes.

In addition to this, you can pretrain a dictionary on sample data, and then use that when compressing files individually

zstd supports using a dict across compression actions, which is what you'd want for this.

The files start at around 700mb and approach 1.8gb by the end of the series.

I've got 64gb of ram and 10 cores to work with, but it seems single core compression might be in order?

7zip can use 2 cores (maybe more now), zstd can use all of them, but zstd doesn't support solid compression - you need to use tar with it.

> Does there exist a compression that could better handle this sort of expanding documents?

Yes: storing deltas. Git might work well. A backup system that does subfile deduplication may work too (e.g. Restic, Borg).

That would be my idea, store the files in git and then backup the whole git repository.

They mention that the files start at 700mb: https://news.ycombinator.com/item?id=33646345

I was under the impression that git didn't do well with large files.

Yeah. We used to store PowerDesigner files in git a while back, which are XMLs usually smaller than 100 MB, and that was pretty much a disaster of hour taking clones already. I can't imagine 700 MB working.

Have you considered git LFS, i.e., git Large File Storage?

I've used it to store 20-80GB of files in a repo and it works quite well.

I'm pretty sure that was with LFS. Maybe the issue was Bitbucket? I'm not sure.

Good point - I wonder if the XML really is "diffable" or if it's basically encoded binary data and has to be stored as a blob anyway.

To expand on the dedup a bit, Borg uses content aware chunking, so any large enough repeats will be fully deduplicated without relying on any specific block size/alignment. This is amazing for "slightly changing files".

lrzip

Long Range ZIP or LZMA RZIP

https://github.com/ckolivas/lrzip

"A compression utility that excels at compressing large files (usually > 10-50 MB). Larger files and/or more free RAM means that the utility will be able to more effectively compress your files (ie: faster / smaller size), especially if the filesize(s) exceed 100 MB. You can either choose to optimise for speed (fast compression / decompression) or size, but not both."

A while back I switched from lz4 to zstd compression with borg for backups. I've always appreciated zstd but the deduplication of borg + zstd is such a dramatic difference in storage space. My home "server" is a tiny NUC-like AMD Ryzen 7 4800U and it flies with this configuration.

Impressive!

The biggest compression you can typically get is if you use structural sharing between backup versions, e.g. using hardlinks or specific filesystem support.

Sorry: if there is absolute maximum bit entropy that cannot be reduced to a generative function, then there can be no (lossless) compression.

This is like the claims of tape drive manufacturers who claimed double the capacity of the media.

Pure marketing wank.

If you're not already compressing and encrypting your backups, then you're not doing it right to begin with. zstd may not be the most efficient algorithm for a given dataset. There are hundreds of compressors out there. It's worth finding the best one on average for a given job and occasionally revalidating it against other candidates.

PSA: Untested backups aren't backups.

I think that you and I have pretty different priors for p(x is a Kolmogorov-random string | x is being backed up)

Zstd Rocks. Probably best Universal compression algorithm. Yes LZ4 can be faster, Brotl can offer better compression but all have other tradeoffs.

Started switching from lz4 to zstd on my ZFS datasets and have seen good results so far. Good stuff.

I don't say this often but... thank you facebook.

https://facebook.github.io/zstd/

This and Presto are about the only things that Facebook did that benefited the world.

Maybe React too, I'm not a front-end person.

“All right, but apart from the sanitation, the medicine, education, wine, public order, irrigation, roads, a fresh water system, and public health, what have the Romans ever done for us?”

Pros: zstd, presto, react

Cons: created a post-truth world which destroyed democracy and killed millions of people by spreading antivaxx propaganda

Eh.

So how does this compare to Twitter, I wonder?

I can't think of any real pros to Twitter. Did they create any great new open-source technologies?

I think Twitter pioneered some pretty cool streaming technologies in its day.

Now Airbnb, those people caused untold suffering by both their impact on housing market, and by creating Airflow ...

I hate to break this to you, but the whole ‘post truth world’ and antivaxx BS existed long before Facebook.

You might as well be blaming highways and radio - Facebook made it more accessible, but far from created it.

Their research in transformers for conversational AI is good. See https://parl.ai/

So far, the societal benefits from Facebook, WhatsApp, and Instagram have been net positive, because the massive amounts of data they're collecting have not yet been bulk misused for mass torture, society-wide oppression, or a large-scale war.

The game isn't over yet, however.

Not in the West, but check Facebook and Myanmar, for example.

The initial release of zstd by Yann Collet was in January 2015 and he started working for Facebook in June 2015.

...compared to LZ4, not exactly honest here, LZ4 on average have worse compression than GZIP, it's just faster.

We actually went back (SQL backups, not xtrabackup) because weirdly enough bzip2 made smaller files when we aimed for compression ratio with similar compress time as bzip2. THink zstd got a bit faster since then tho...

Only for MySQL, PostgreSQL were noticeably smaller with zstd

Reply on news.ycombinator.com