Show HN: Sshync – Auto-sync files over SSH | Hacker News Reader

Show HN: Sshync – Auto-sync files over SSH

github.com

65 pointsmegalodon53 comments

This is a nice little project, but I think rsync is better suited for any serious use. It can do ssh, it's own protocol or any connection program that would tunnel bidirectionally. It does very efficient transfer, can create off-line patch files, and basically do any one-way file/dir sync function you can think of.

Edit: this one is continuous and rsync is one-shot. Personally, I've been using inotify+rsync+timeout (intofy+rsync has a race condition, so limiting the inotify wait and syncing anyway is essential). Does sshync avoid the race?

Yes it combines also amazingly well with inotifywait[1], perhaps there is a port for other OS-es too.

Here is a tip on how it can be used ( not tested )

    #!/bin/bash

    WATCH_PATH="/some/path"
    DEST_PATH="/some/path"
    REMOTE="root@example.com"

    while read LINE; do
        FILE=$(echo ${LINE} | cut -d' ' -f2 )
        /usr/bin/rsync -r ${FILE} ${REMOTE}:${DEST_PATH}
    done < <(/usr/bin/inotifywait -mq -e CLOSE_WRITE --format "%w %f" ${WATCH_PATH})

1 : http://linux.die.net/man/1/inotifywait

For most uses, rsync -a (preserve all possible metadata, like cp -a) is preferable to rsync -r ;

Also, unless you have a very large (breadth or depth) tree, you can just let rsync synchronize the entire path - it is quite efficient, will pick up additional diretories (you need inotifywait -r for that, and AFAIK you still have an unavoidable race).

If you're considering replacing your current (assumedly homegrown) solution; I'd recommend using lsyncd instead.

You might also check out Syncany if you're interested in client-side encryption or Syncthing if you're looking for something more Bittorrent Sync like.

I use rsync. Question for you (or any expert here). Is there any possible way that rsync can alter the source file (unintentionally I mean, by accident or bug)?

And what happens to an rsync file that is modified while in the middle of a transfer? [1](I haven't been able to find satisfactory answers to these questions hence the question here).

[1] Say you have a 1,000,000 line file that you are rsynch-ing and you delete line 50000 and insert at line 750000?

Rsync opens source file in read only mode - so it has the same probability to modify a file that regular copying does. That said, the --remove-source-files option will delete source files after copying; --delete may remove destination files; and used together they drop the idempotence guarantee that rsync gives. So you have to be careful.

Re changing file - IIRC it will keep Resending until file size+mtime at end of transfer is same as in the beginning.

I use similar solution: my own shell script with inotify + timeout + unison over ssh. Unison is much better suited for two-way synchronization than rsync.

Don't know if this is still the case, but in the past, unison was extremely picky about the versions at both ends; I stopped using it when I upgraded my Ubuntu box and as a result I needed to either recompile an older version for the new box or upgrade the version on all other boxes (across windows and various distros).

I now use rsync for bidirectional mirroring (any two versions from the last 15 years or so can talk to each other), git for synchronization (getting history for the price of needing to add/commit). Can't beat the network efficiency and compatibility of these two.

For http://www.cis.upenn.edu/~bcpierce/unison/ your thoughts on why that is better than using rsync ?

Edit: I am now noticing this tool isn't even maintained anymore.

From https://alliance.seas.upenn.edu/~bcpierce/wiki/index.php?n=M...

"Rsync is a mirroring tool; Unison is a synchronizer. That is, rsync needs to be told "this replica contains the true versions of all the files; please make the other replica look exactly the same." Unison is capable of recognizing updates in both replicas and deciding which way they should be propagated."

Larry - would you please email us at info@rsync.net ? We have a few items to discuss with you ...

(your profile mentions to respond to one of your comments if someone needs to reach you ...)

Thanks - I will but not just yet (a few reasons for that actually).

A new version has been released this year. unison works remarkably well, it's understandable there is not much to add.

I love unison, but how do you use it with inotify?

Newer versions of Unison (since around 2012) can watch directories with inotify: http://blog.nguyenvq.com/blog/2011/07/18/real-time-file-sync...

These are some pretty awful scripts, to be honest. Use Unison (https://www.cis.upenn.edu/~bcpierce/unison/). Syncing is much more dangerous and difficult than people assume.

There is a standard tool, ssh-copy-id, that does the job of sshpair.

This sshpair is actually quite dangerous as it simply overwrites your existing key.

Yeah I was just complaining about that elsewhere. It's a further symptom of the fact this whole project is reinventing wheels, badly.

* Standard on Debian based OSes

It's literally just a shell script, it's not like it can't be ported to other *nixes

http://anonscm.debian.org/cgit/pkg-ssh/openssh.git/tree/cont...

    $ rpm -qf /usr/bin/ssh-copy-id 
    openssh-clients-6.9p1-6.fc22.1.x86_64

Yes, its actually in the contrib directory in the openssh source (I didn't realise this until I just checked)

https://anongit.mindrot.org/openssh.git/tree/contrib/ssh-cop...

Hmmm, yes, it seems it is more generally distributed now than it was in the past. I just found it's on a lot more distributions than I had recalled.

This is good to learn!

I use lsyncd for this. Why would I switch?

Me too. Lsyncd rules! I use it to edit locally and run the code in an AWS instance. I even wrote something about it in my blog.

http://aurelianito.blogspot.com.ar/2014/12/mejorando-el-mirr...

(We do as well). From what I see, the only advantage this offers over lsyncd is that Windows is supported.

Maybe because lsyncd have nothing update from a year https://github.com/axkibe/lsyncd

This project relies on the npm module "ssh-client" which is at v0.0.1 and hasn't been updated for three years.

Want to compare any other useless statistics?

There's also sshfs: https://en.wikipedia.org/wiki/SSHFS

Note however that sshfs makes a file system available through ssh (with Some caching but not e.g. persistent cache across ssh reconnects), and no off-line persistence - whereas other solutions discussed (sshync, unison, lsync, syncthing) all replicate files (think dropbox) which comes with benefits (quicker access, offline availability) and problems (edit conflicts, stale versions)

Hey.

I appreciate the positive response, but this is just something I put together quickly and needs a little refurbishing before it can be considered a viable alternative to something like lsyncd.

Top priorities right now are to get rid of ssh-client dependency and to address security concerns. I'd also like to implement file diffing.

Your feedback is invaluable!

This is a small thing - but how do you pronounce the name? Looking at it, I want to call it "ssh sync", but it looks more like it should sound like "shync" (which sounds sort of like Garth from "Wayne's World" saying "schwing!") Just curious because it's hard to discuss a tool verbally if I don't know how to pronounce it.

I'd go with "ssh sync".

This is how I do my email, exim running on a DO instance and I send and receive to my laptop via ssh.

On Windows I use WinSCP, from menu "Commands\Synchronize" and "Commands\Keep Remote Directory up to Date", or from scripting "synchronize" and "keepuptodate".

Why not use the existing rsync over ssh and ssh-copy-id?

I think newline is valid in filenames in most *nix filesystems, so the possibility to use an alternative RS would be nice.

What's difference between this one and rsync？

Javascript!

And fewer features.

(but a cool little project none the less!)

It looks to me that this keeps running "in the background" and watches for changes; syncing things as they happen. Rsync can't do that.

rsync can however take a file list, and inotifywait can produce a file list with the help of awk or sed. Which means you can achieve continuous synchronisation with ~5 lines of shell script. A bit more care if you want to sync two ways.

I've found that my file syncing needs tends to be so varied that most of the dedicated tools I've looked at have almost always been just wrong enough not to be worth it compared to just composing something with rsync and inotifywait.

Of course your mileage will vary.

True but you could spawn rsync with a different tool that uses inotify, for example: http://git.quitesimple.org/adhocify/tree/README

rsync is not written in a hipster language.

This is 130 lines of code, including whitespace.

Plus node, and npm. That's an extra 59 packages from Apt. before you install this tool.

So we will roll glibc and everything else into the counts too?

My point was it isn't "hipster" to use something that is fast and efficient.

Writing yet another file sync solution using nodejs of all things is definitely hipster.

Glibc is used by multiple packages. I simply identified the number of dependencies to install this tool when nodejs/npm are not installed.

https://github.com/mattijs/node-rsync

You can use rsync if you like too

this works on windows

rsync can be run in cygwin.

Quoting beagle3

> this one is continuous and rsync is one-shot.

I pronounced this 's-shync' in my head. I am happy.

Reply on news.ycombinator.com