greaber

As for speed, see https://greaber.github.io/syq-bench/rclone.html. As for other stuff, it depends what you use rclone for.

I did some benchmarks against rclone today. https://greaber.github.io/syq-bench/rclone.html Syq was sometimes much faster and never more than a few percent slower. I didn't do any tuning of syq since it is meant to perform well without any special options, but I did tune a couple of rclone options. It's possible that more extensive rclone tuning would yield better results though. Please let me know if there is anything I need to try!

Syq has compression turned on by default. Also, if you are updating a file that already exists on the other end, syq will only send blocks of the file that changed.

Also, in syq, a connection is the unit of independent transfer work. Probably it would be possible to get higher bandwidth per TCP connection by some kind of multiplexing, but that architecture would be more complex and probably not have many benefits.

They are benchmarks on real hardware, and I tried to provide enough details so that anyone can reproduce them (but some of the benchmarks are on servers I have that you can't order on public cloud -- these ones are clearly marked). "Synthetic" means that the data is synthetic, not an actual production workload. For instance, some of the benchmarks use files filled with random bytes.

Nice example. Rsync might be better here. Although syq will still avoid copying most of the image if changing the config file just modifies a small number of locations and doesn't globally move data around (like prepending a single byte to the ISO)

Yeah, I kind of regret writing "better than rsync" in the tagline. Maybe I should have written "better than rsync in some respects".

Yeah, I get that. I really didn't mean to sound conceited, and I don't think that syq is better than rsync for every use case. I was just trying to convey in a very short space that it is something that rsync users might be interested in because it solves some frustrations with rsync.

Thanks! Yeah, I was a little surprised (maybe I shouldn't have been) at how negative some of the comments were. I will think about what additional explanations I can provide that will help people. Regarding rclone, I have also spent a bunch of time looking at faster transfers to and from S3-compatible storage, but that problem is somewhat different, and I didn't look in depth at how rclone is implemented. FWIW, in my experience, s5cmd is generally a bit faster than rclone even if you tune rclone options. (But rclone is more flexible.) My guess is that s5cmd is already close to the performance ceiling, but I don't really know.

As for edge cases, I will also try to document that more. In general, when I'm not sure what semantics to go for, I try to either copy what rsync does or do something safer. For instance, one thing I am looking at now is the best way to handle cases where a source path cannot be represented on the target filesystem or where two source paths would collide (e.g. due to unicode normalization or case insensitivity). Tentatively, my inclination is to try to fail before copying any files when possible. Currently, syq doesn't do this (but neither does rsync).

I entirely agree that multiple connections shouldn't actually be necessary for speed, but they help in multiple situations (long-distance transfers, same DC between servers, NFS), and it's not just about amortizing per-file setup overhead.

There is some info on the optimizations in the docs, but I agree that a more complete technical explanation of all the things syq does could be useful. I will work on one. On the other hand, I also tried hard to make it just go fast without needing the user to understand why it is fast or tune anything. For instance, the number of connections is auto-tuned by default.

Yeah, ControlMaster is cool and was an inspiration, but since the syq client is always talking to the same receiver on the server, we can save a few round trips that ControlMaster still needs, so syq persist feels faster. This architecture is also what enables reverse mode, where you can download to your laptop while working on the server.

ProxyJump doesn't help when you want to copy files from server A to server B without giving server A an agent that can do arbitrary things on server B. There is really no alternative to using a restricted authorized key on server B for this scenario.

Currently, there is just a fixed range of ports that it will use if they are open and you don't pass `--no-tcp`. It does its own encryption over the TCP connections. Also worth knowing is that for some connections (e.g. if there are any dropped packets), you may get much better performance if you can enable BBR congestion control. https://greaber.github.io/syq/server-tuning.html#test-conges...

It just installs in $HOME/.local/bin by default (and some other parts go in other locations inside your home directory). You don't need superuser access for anything. The first time you connect to a server, it installs its matching counterpart in your home directory on the server.

I'm seeing 5x speedups on my real workloads, and you can see some synthetic benchmarks here or run your own https://greaber.github.io/syq-bench/

Thanks for the suggestion! I'm not very familiar with btrfs send and btrfs receive. Browsing the documentation, it looks cool, but maybe you could say more about the specific things that make it so much better for your use case? Could any of them be incorporated into a more general tool like syq? Also, it doesn't look like btrfs send helps transfers go fast by parallelizing them or has anything like syq's remote-remote transfer feature, which made me wonder if it would actually make sense to make syq able to work as an authenticated fast transport for a stream generated by btrfs send/receive.

- No need to be snarky. It's a new project, while rsync is time tested, and that is certainly an advantage for rsync and all old software.

- You can install by piping curl to bash, using brew, or by compiling yourself. One thing about distribution I would argue syq gets right is that you are never relying on whatever version of syq happens to be installed on a server. The client syq always talks to a server syq tagged with the exact same version, and when your local copy of syq installs a remote copy, it verifies that the remote binary is signed by me.

- I never said or implied that rsync has a byzantine interface or poor documentation. I think rsync is well documented and the interface is overall fine. Syq even has an rsync compatibility mode, so you don't have to learn any new syntax if you don't want. Still, I tried to provide good docs for syq too, and honestly even if there were a gap in the docs, all you have to do these days is ask AI to look at the source code and tell you what to do. I think this also mitigates the trust issue with a new project from a single contributor.

There are a bunch of reasons, some of which are explained in the description and in the docs. For me, the most important benefit is speed of copying. Note that syq doesn't currently implement rsync's delta merge algorithm, which allows it to avoid copying data that is already in the destination file but at a shifted offset. I might implement this (or an enhanced version of it) in the future. If your workloads have a lot of cases like this then syq might not be better than rsync for you, but I found that for me this rarely came up.

Doesn't this formula also punish circular trade patterns in a strange way, like if the US imports $X of good from country B and country B imports $X of goods from country C and country C imports $X of goods from the US, and there are no flows in the opposite direction, then the US would impose an infinite tariff on country B since US imports from B are zero?

How does CapRover work with databases? Does it replace something like RDS?