Hacker News Reader

Comment by Dunedan | Hacker News Reader

I wonder if the at least partially the reason for the speed up isn't the multi-threading, but instead that rclone maybe doesn't compress transferred data by default. That's what rsync does when using SSH, so for already compressed data (like videos for example) disabling SSH compression when invoking rsync speeds it up significantly:

  rsync -e "ssh -o Compression=no" ...

View on HN

Compression is off by default in OpenSSH, at least `man 5 ssh_config` says:

> Specifies whether to use compression. The argument must be yes or no (the default).

So I'm surprised you see speedups with your invocation.

Good point. Seems like I enabled it in ~/.ssh/config ages ago and did forget about it. Nonetheless, it's good to check whether it's enabled when using rsync to transfer large, already well compressed files.

IIRC rsync uses your default SSH options, so turning off compression is only needed if your default config explicitly turns it on (generally or just for that host). If sending compressible content using rsync's compression instead of SSH's is more effective when updating files because even if not sending everything it can use it to form the compression dictionary window for what does get sent (though for sending whoe files, SSH's compression may be preferable as rsync is single threaded and using SSH's compression moves that chunk of work to the SSH process).

Reply on news.ycombinator.com