Hacker News Reader

> Does there exist a compression that could better handle this sort of expanding documents?

Yes: storing deltas. Git might work well. A backup system that does subfile deduplication may work too (e.g. Restic, Borg).

View on HN

To expand on the dedup a bit, Borg uses content aware chunking, so any large enough repeats will be fully deduplicated without relying on any specific block size/alignment. This is amazing for "slightly changing files".

That would be my idea, store the files in git and then backup the whole git repository.

They mention that the files start at 700mb: https://news.ycombinator.com/item?id=33646345

I was under the impression that git didn't do well with large files.

Yeah. We used to store PowerDesigner files in git a while back, which are XMLs usually smaller than 100 MB, and that was pretty much a disaster of hour taking clones already. I can't imagine 700 MB working.

Have you considered git LFS, i.e., git Large File Storage?

I've used it to store 20-80GB of files in a repo and it works quite well.

I'm pretty sure that was with LFS. Maybe the issue was Bitbucket? I'm not sure.

Good point - I wonder if the XML really is "diffable" or if it's basically encoded binary data and has to be stored as a blob anyway.

Reply on news.ycombinator.com