Show HN: Orgalorg – modern parallel SSH, file sync and cluster maintenance tool

By implementing your own ssh, you appear to be ignoring lots of useful default ssh utilities, like ssh-agent and ~/.ssh/config eg.

    [derf@pan][09:06:20]-[~] $ orgalorg -o pandora -C uptime
    2016-07-25 09:06:28 [FATAL] can't create runner factory
    └─ can't read SSH key: '/Users/derf/.ssh/id_rsa'
       └─ can't read SSH key from file
          └─ open /Users/derf/.ssh/id_rsa: no such file or directory
    [derf@pan][09:06:28]-[~] $ orgalorg -p -o pandora -C uptime
    Password:
    2016-07-25 09:06:36  [WARN] still connecting to address after 2s: [derf@pandora:22]
    2016-07-25 09:06:44 [FATAL] acquiring global cluster lock failed
    └─ connection to 1 of 1 nodes failed
       └─ can't connect to nodes
          └─ can't connect to address: [derf@pandora:22]
             └─ dial tcp 18.60.0.124:22: i/o timeout
    [derf@pan][09:06:45]-[~] $ ssh pandora uptime
     09:06:49 up 37 days, 23:45,  0 users,  load average: 1.18, 0.70, 0.33

Is the assumption that I'm going to set up entries in /etc/hosts for all my systems, or add them to my network's DNS? What about ssh proxy hosts? The fact that you require a single ssh key to live in .ssh/id_rsa is going to make this useless for anyone that does any advanced usage of SSH.

Don't forget about fabric http://www.fabfile.org/ Older, much more tested, pythonic blah blah blah

DisclaimerAsImAmericanAndIDisclaimEverything: I'm not author. Not even user. I just know it exists.

For the 'Alternatives' section, there's also gnu parallel which I have been fairly happy with.

https://www.gnu.org/software/parallel/man.html#DIFFERENCES-B...

This looks like a really nice tool.

Also +1 for listing some alternatives and comparisons in the README, so that we don't have to bitch "but you should use $X instead" here :-)

This tool found it's niche as there were no reliable way to do parallel jobs with just one command.

Reminds me of one of my first Python projects:

https://pypi.python.org/pypi/sshpt

"SSH Power Tool, Run commands and copy files to multiple servers simultaneously WITHOUT requiring pre-shared authentication keys"

These days I figured everyone that wanted to do such things would use a tool like Ansible.

First of all, congratulations on getting this to a point where you can show it off. A lot of hard work clearly went into this, and it's can't be an easy decision to unleash the hordes of HN programmers knowing the inevitable "constructive criticism" that results. I think a lot of the people who ask "why not just use X" might seem like they miss the point of writing a project to do something you need done just for the fun of it, but I'm an optimist so I assume they're just interested.

I'm an HPC sysadmin, so I use tools like this (specifically pdsh+pdcp) every day. Automation is great and we use it for just about everything, but the nature of the beast is that there's a lot of ad-hoc work and testing that needs to be done.

Specifically with respect to pdsh, just because it appears to have been abandoned, that doesn't necessarily make it the wrong tool for the job. It's pretty-well feature-complete for everything I use it for (that said, it was designed precisely for my needs, which likely aren't yours).

Some minor practical features that would make it a worthy replacement for the venerable pdsh:

  * range selection in the hostname selection going beyond bash expansion, like 
    -o compute[0012-0034,1123,1156-1199]
  * taking hostnames from a file is a must
  * supporting libgenders would be amazing

Nice work! I'll try it out and file github issues.

The examples in the GIF-animation can be emulated by GNU Parallel using this:

To make the emulation easier, make a simple alias:

  alias par_emul="parallel -j0 --ssh 'sshpass ssh' --nonall --tag --linebuffer"

If you want to supply a password run:

  SSHPASS=`ssh-askpass`

or set the password directly:

  SSHPASS=P4$$w0rd!

If the above is set up you can then do:

  orgalorg -o frontend1 -o frontend2 -p -C uptime
  par_emul -S frontend1 -S frontend2 uptime

  orgalorg -o frontend1 -o frontend2 -p -C top -bid 1
  par_emul -S frontend1 -S frontend2 top -bid 1
                                                                                            
  orgalorg -o frontend1 -o frontend2 -p -er /tmp -n 'md5sum /tmp/bigfile' -S bigfile
  par_emul -S frontend1 -S frontend2 --basefile bigfile --workdir /tmp  md5sum /tmp/bigfile

There are a few differences:

* GNU Parallel normally uses 'ssh' and recommends using 'ssh-agent'.

* With 'sshpass' you only have to type the password once.

* GNU Parallel does not have progress indication for the file transfer.

* GNU Parallel uses /./ to indicate which part of the path should be copied to the remote side: foo/bar/./sub/dir/file will append sub/dir/file to --workdir on the remote side.

I use Rex for this (http://rexify.org/).

For example:

rex -H "frontend[01..15]" -e "say run 'uptime'"

Another alternative

https://github.com/square/mssh

What's the added value over xargs + ssh? Eg:

    echo host1 host2 |
    xargs -P2 -n1 -I% ssh % uptime

if its just a single command, parallel[1] works great:

    parallel --tag 'ssh {} uptime' ::: host{1..10}

1. https://www.gnu.org/software/parallel/

While a lot of this functionality already exists, this tool is great in combining it all.

P.S: not a big fan of the name :p

We built something similar to this a while back: https://github.com/Mojang/Sift, it's great to see more alternatives popping up, seems like a great tool :)

Off-Topic: What's the shell/config you're using in the README gif ? It looks damn cool.

Can't help but quote: "If you have to SSH to your production box, your automation has failed".

This tool could be a nice thing to tinker and experiment with a cluster setup, though.

You don't support GSSAPI?

It's always good to see people challenging existing solutions .. this is how innovation happens. But you need to mention other good alternatives like fabric, paramiko, etc to list of alternatives... and plain old shell scripting.

Some of the other options in orgalog seem like a combination of rsync, autossh (which has a heartbeat protocol built in), xargs, and plain old bash might be a better option.

For ad hoc jobs like this, this seems like how I'd go about synchronizing local and remote directories on multiple servers in parallel:

    for host in host1 host2 host3
    do
         rsync -rpae ssh /my/directory/ $host:/remote/directory/ &
    done
    wait

That will synchronize a local directory with a remote directory, in parallel, and it's readable and uses software you probably already have installed.

Or, if you want to limit parallelization to no more than 16 boxes simultaneously, use in combination with xargs (ie something like xargs --max-procs 16 -n 1 and call a short bash function with the given hostname). Orgalog doesn't seem to support limiting to n hosts at a time?

Here's how to run the example uptime on hosts in parallel, but with batch limits:

orgalog:

    orgalorg -o <host>... -x -C whoami

bash:

    echo host1 host2 ... | xargs -d' ' -P2 -n1 -I{}     ssh {} uptime

Another orgalog example:

    orgalorg -o <host>... -p -i ~/.ssh/id_rsa.pub -C tee -a ~/.ssh/authorized_keys

bash:

    echo host1 host2 ... | xargs -d' ' -P2 -n1 -I{} scp ~/.ssh/authorized_keys {}
    # or, Userify cloud or On Site for >1 users

Of course, I'd just use Userify[1] in a local installation for that (disclaimer: CTO) but you could just modify the rsync example above or just use my old cloudadmins script[2] if your needs are small.

This was obviously a big effort just looking at the dozens of go files! I don't want to disparage what was obviously a huge amount of work from someone who spent a lot of time. A cautionary note for anyone thinking about this: always focus on automating non-security-sensitive tasks first and try to use the smallest amount of magic necessary to do a trick. Rolling your own crypto is never a good idea.

Clearly the OP is trying a lot of new ideas and we should expect good things in the future.

1. https://userify.com

2. https://github.com/jamiesonbecker/cloudadmins