Hacker News Reader

Comment by mromanuk | Hacker News Reader

I was surprised that he didn't try to use on the flight compression, provided by rsync:

  -z, --compress              compress file data during the transfer
      --compress-level=NUM    explicitly set compression level

Probably it's faster to compress to gzip and later transfer. But it's nice to have the possibility to improve the transfer with a a flag.

View on HN

Or better yet, since they cite corruption issues, sqlite3_rsync (https://sqlite.org/rsync.html) with -z

sqlite transaction- and WAL-aware rsync with inflight compression.

The main point is to skip the indices, which you have to do pre-compression.

When I do stuff like this, I stream the dump straight into gzip. (You can usually figure out a way to stream directly to the destination without an intermediate file at all.)

Plus this way it stays stored compressed at its destination. If your purpose is backup rather than a poor man's replication.

The main point was decreasing the transfer time - if rsync -z makes it short enough, it doesn't matter if the indices are there or not, and you also skip the step of re-creating the DB from the text file.

The point of the article is that it does matter if the indices are there. And indices generally don't compress very well anyways. What compresses well are usually things like human-readable text fields or booleans/enums.

I believe compression is only good on slow speed networks.

Valve tends to take a different view...

Valve has different needs then most. Their files are rarely change so they only need to do expensive compression once and they save a ton in bandwidth/storage along with fact that their users are more tolerant of download responsiveness.

Depends. Run a benchmark on your own hardware/network. ZFS uses in-flight compression because CPUs are generally faster than disks. That may or may not be the case for your setup.

It would have to be one really fast network... zstd compresses and decompresses at 5+ GB (bytes, not bits) per second.

Where are you getting this performance? On the average computer this is by far not the speed.

I just tested on a ramdisk:

  tool  cspeed    size  dspeed
  zstd  361 MB/s  16%   1321 MB/s
  lzop  512 MB/s  29%    539 MB/s
  lz4   555 MB/s  29%   1190 MB/s

If working from files on disk that happen not to be cached, the speed differences are likely to disappear, even on many NVMe disks.

(It just so happens that the concatenation of all text-looking .tar files I happen to have on this machine is roughly a gigabyte (though I did the math for the actual size)).

Looks like it depends heavily on choice of file, but I see good performance on both compressible and uncompressible files. Small files tend to perform (relatively) bad though. Here is a sample of 3 large files with different compression ratios:

  zstd -b1 --fast -i10 some-rpi-os-image-idk.img
  -1#-os-image-idk.img :2097152000 -> 226798302 (x9.247), 6765.0 MB/s, 5897.3 MB/s

  zstd -b1 --fast -i10 jdk-11.0.8+10.tar
  -1#jdk-11.0.8+10.tar : 688844800 -> 142114709 (x4.847), 2660.7 MB/s, 2506.8 MB/s

  zstd -b1 --fast -i10 archlinux-2025.04.01-x86_64.iso
  -1#.04.01-x86_64.iso :1236303872 ->1221271223 (x1.012), 3643.5 MB/s, 7836.6 MB/s

Ain't no way zstd compresses at 5+, even at -1. That's the sort of throughputs you see on lz4 running on a bunch of core (either half a dozen very fast, or 12~16 merely fast).

Is the network only doing an rsync? Then you are probably right.

For every other network, you should compress as you are likely dealing with multiple tenants that would all like a piece of your 40Gbps bandwidth.

In your logic, you should not compress as multiple tenants would all like a piece of your CPU.

This will always be something you have to determine for your own situation. At least at my work, CPU cores are plentiful, IO isn't. We rarely have apps that need more than a fraction of the CPU cores (barring garbage collection). Yet we are often serving fairly large chunks of data from those same apps.

What? Compression is absolutely essential throughout computing as a whole, especially as CPUs have gotten faster. If you have compressible data sent over the network (or even on disk / in RAM) there's a good chance you should be compressing it. Faster links have not undercut this reality in any significant way.

Whether or not to compress data before transfer is VERY situationally dependent. I have seen it go both ways and the real-world results do not not always match intuition. At the end of the day, if you care about performance, you still have to do proper testing.

(This is the same spiel I give whenever someone says swap on Linux is or is not always beneficial.)

He absolutely should be doing this, because by using rsync on a compressed file he's passing by the whole point of using rsync, which is the rolling-checksum based algorithm that allows to transfer diffs.

or used --remove-source-files so they didn't have to ssh back to rm

Reply on news.ycombinator.com