Hacker News Reader

Comment by donatj | Hacker News Reader

So we've got a system that backs up to an XML file nightly. I noticed a couple weeks ago the bucket was using a not-insignificant amount of disk space.

These XML files are full backups rather than deltas - so each one contains the full previous file plus the new additional data.

My assumption was if I grabbed just a swath of the old ones, say a years worth and compressed them together I would get really decent savings.

I was pretty disappointed - gzip knocked 10 gb off of about 100 gb of data. I started doing some research and found people saying 7zip and it's sliding dictionary size options were the answer. After multiple tries, each run of 7zip taking multiple days I was able to get it down to about 70 gigabytes from 100. Better then gzip but frankly nowhere near what I would expect.

Does there exist a compression that could better handle this sort of expanding documents?

View on HN

You need what's called "solid compression", meaning that the files are concatenated together before compression. Because otherwise compression restarts from zero for each new file - https://en.wikipedia.org/wiki/Solid_compression

Some compressors have options for solid compression, or you can use tar to first concatenate the files. For both, you need to sort the files first, so that they are compressed in chronological order, otherwise the common information will drop out of the dictionary.

And how big is one file and how much RAM you have? Because you might need increasing the "dictionary size" of the compression algorithm, both 7zip and zstd support multi-gigabyte sizes.

In addition to this, you can pretrain a dictionary on sample data, and then use that when compressing files individually

zstd supports using a dict across compression actions, which is what you'd want for this.

The files start at around 700mb and approach 1.8gb by the end of the series.

I've got 64gb of ram and 10 cores to work with, but it seems single core compression might be in order?

7zip can use 2 cores (maybe more now), zstd can use all of them, but zstd doesn't support solid compression - you need to use tar with it.

> Does there exist a compression that could better handle this sort of expanding documents?

Yes: storing deltas. Git might work well. A backup system that does subfile deduplication may work too (e.g. Restic, Borg).

That would be my idea, store the files in git and then backup the whole git repository.

They mention that the files start at 700mb: https://news.ycombinator.com/item?id=33646345

I was under the impression that git didn't do well with large files.

Yeah. We used to store PowerDesigner files in git a while back, which are XMLs usually smaller than 100 MB, and that was pretty much a disaster of hour taking clones already. I can't imagine 700 MB working.

Have you considered git LFS, i.e., git Large File Storage?

I've used it to store 20-80GB of files in a repo and it works quite well.

I'm pretty sure that was with LFS. Maybe the issue was Bitbucket? I'm not sure.

Good point - I wonder if the XML really is "diffable" or if it's basically encoded binary data and has to be stored as a blob anyway.

To expand on the dedup a bit, Borg uses content aware chunking, so any large enough repeats will be fully deduplicated without relying on any specific block size/alignment. This is amazing for "slightly changing files".

lrzip

Long Range ZIP or LZMA RZIP

https://github.com/ckolivas/lrzip

"A compression utility that excels at compressing large files (usually > 10-50 MB). Larger files and/or more free RAM means that the utility will be able to more effectively compress your files (ie: faster / smaller size), especially if the filesize(s) exceed 100 MB. You can either choose to optimise for speed (fast compression / decompression) or size, but not both."

That doesn’t make much sense, window size or not. Even base64 encoded random data would be about that bad.

Is the XML wrapping a bunch of other random data or something?

base64 random data would be that bad only because of base64. Random data does not compress at all on average.

As an example, I have zstd enabled on some zfs pool. The client-side encrypted time machine backups does not even compress 1 %, as expected.

Yes, that’s why I said that?

You can’t embed binary data in xml raw, so a common pattern is embedding it as base64, or similar type of wrapping.

Technically, it’s possible to escape it in other ways, but it’s error prone.

Either way, XML which has text strings or whatever typical XML document data that ISN’T something like base64 encoded random data should compress dramatically better than what the poster was talking about.

So, what the hell is in your XML anyway?

Ah sorry, I misunderstood what you said

Thanks for responding. Apologies for the cursing to add excessive flair.

Reply on news.ycombinator.com