Cluster SSH (OSX) | Hacker News Reader

Cluster SSH (OSX)

code.google.com

124 pointsxtacy58 comments

Also I'd recommend kanif:

http://taktuk.gforge.inria.fr/kanif/

This is a great tool to do mass changes to our 1200 CPU compute cluster. The best part about this is, it will map all of the hosts with the same output together and reduce this down to just one instance of the output, with the affected hostnames listed together. So instead of 500 instances of the same output data I get one. This comes in handy for finding hosts which are different than what I'm looking for.

Simple example:

  --------------------------------------------------------------------------------
  compute[0009,0051,0187] (3 HOSTS)
  --------------------------------------------------------------------------------
  Filesystem            Size  Used Avail Use% Mounted on
  /dev/mapper/rootvg-rootlv
                         58G   14G   42G  25% /
  --------------------------------------------------------------------------------
  compute[0013,0038,0044,0046,0065,0103,0125,0142,0195,0213,0216,0234] (12 HOSTS)
  --------------------------------------------------------------------------------
  Filesystem            Size  Used Avail Use% Mounted on
  /dev/mapper/rootvg-rootlv
                         58G   11G   44G  19% /
  --------------------------------------------------------------------------------
    compute0247 (1 HOST)
  --------------------------------------------------------------------------------
  Filesystem            Size  Used Avail Use% Mounted on
  /dev/mapper/rootvg-rootlv
                       58G  7.6G   47G  15% /

Many companies manage their small clusters mostly by hand, even if they know there should be a better way. You just showed me one better way--thanks!

Cluster SSH is a terrific tool. When I learned of it I coded up a quick python script to play selected tracks of MIDI files and used CSSH to play Bach's Toccata and Fugue in D minor polyphonically on ~20 computers' PC speakers.

It was also fun to terrify first year students with 20 cd trays ejecting at the same time...

isn't ejecting cd trays so, like, 90's :) glad my mac book doesn't have one!

This is surely the most elaborate excuse ever for not writing those maintenance scripts?

It's great for those little tweaks like adding a symbolic link.

meh...

    for h in host1 host2 host3; do \
        echo -e "\n$h\n===========\n"; \
        ssh $h 'ln -s /some/path /path/to/link && ls -l /path/to/link'; \
    done

edit: woops, left out a couple semicolons...

> edit: woops

Exactly.

One-offs are easy to test interactively with cssh, and despite what anyone thinks, there are always one-offs in the real world.

True. Usually when creating a shell loop like this, I prepend 'echo' in front of 'ssh...' as a dry run first. I also wouldn't actually use something like this for commands that are destructive in nature. Usually it's for things like running informational commands ('rpm -q', 'uptime', etc). Of course anything can become destructive if you fat-finger it hard enough.

In short, use automation with care.

FUCKING SERIOUSLY?!

Thank you for this comment, I never knew you could pass commands to ssh.

i suggest perusing the ssh man, then, as it seems likely that it has other capabilities you don't know about.

Like how SSH agent forwarding can allow you to hop from box to box with the SSH keys loaded onto your local machine's SSH agent? Found that one a few months ago, made me reflect on what I've been doing in life the past 10 years not thoroughly reading UNIX man pages.

Thanks for this one. You just added a few days to my life and helped push out the onset of carpal tunnel/arthritis. ~ http://www.unixwiz.net/techtips/ssh-agent-forwarding.html

ssh is magic. checkout -X for X11 forwarding to run remote GUI apps, and -R which lets you set up a reverse SSH tunnel when you need to connect to a machine behind a firewall.

such a polite version of RTFM, i love you

Based on a similar reaction a few months ago, this will definitely change the way you go about system work.

Now combine that with dsh (http://www.netfort.gr.jp/~dancer/software/dsh.html.en) and authorized_keys, you can _really_ get in trouble. dsh is similar to this Cluster SSH, but is appropriate for use in scripts, cron jobs, etc.

Example:

dsh -Mg all-machines -F 20 'do something'

-M: Prepend any line of output with the machine that emitted it

-g: runs the group all-machines (just a text file in etc/dsh/group)

-F 20: forks at most 20 copies of ssh in the background so you don't overload the server you're running this from

You probably already know this, but if you have a significant number of hosts:

    for i in `seq 1 30`; do \
        echo -e "\nhost$i\n===========\n"; \
        ssh host$i 'ln -s /some/path /path/to/link && ls -l /path/to/link'; \
    done

Also, if you just start typing

  for i in `seq 1 30`
  do
     commands
     more commands
  done

On the command line itself, you don't need to worry about semicolons. Bash will take care of it.

    for i in {1..30}; do...

Ah, I was not aware of seq; good one. :)

Good point about not needing semicolons for multiple-line stuff (assuming the EOL's are not escaped, as you have it in your example). I blame the PHP I spent all week immersed in...bleh.

Could be worthwhile seeing when something goes wrong on one machine. It would be great to bundle all terminals that look the same though

Neat idea, though I feel like it would be better implemented as a command line tool using ncurses or something similar so it could be used on any Unixy system (including logged into another server, for example)

I suppose april fools is the reason we've gotten so far through this discussion that only scattered mentions of fabric or puppet/chef?

I've certainly done my share of ssh in a for loop, but if you're seriously considering this you almost certainly want to look into puppet/chef/cfengine/etc.

Now that <a href="http://devstructure.com>blueprint</a> exists, it's radically simple to turn your current environment into chef/puppet deploy script. That said, distributed shell tools like this are still tremendously useful and should be part of any cluster stack.

tmux equivalent:

  C-b % [n times, n panes]
  C-b M-5 [tiled layout]
  ### ssh into your servers in each pane, or whatever
  C-b : setw sy <tab> [short for "synchronize-panes"]
  ### have synced fun!

This like these always makes me uneasy. Yes, you can do stuff on X number of machines all at the same time. That also means you can break things all at the same time.

Only use I can see if for multi-system stats - which should be handled differently anyhow, but that's a different story. Perhaps running htop, bwm-ng, iftop or other similar tools. Would certainly be scary to do anything other than that...

CSSHX + sudo su is cruise control for cool.

If you run rm -r / you'll learn where they got the name "cluster" from

Thanks; it's always nice to learn of another option.

I've been using sshpt and pssh for this sort of thing:

http://code.google.com/p/sshpt/

http://code.google.com/p/parallel-ssh/

This should look great in movies.

Cluster SSH is also a tool for Emacs: http://www.emacswiki.org/emacs/ClusterSSH

This seems like something that would be much better if integrated into a tool like Fabric or Capistrano that already knows how to send SSH commands to a bunch of different servers.

Does anything like that exist? I guess the downside would be fixing problems if your command failed in 1 or 2 of 10 or 20 servers… Then you'd have to open up a separate session to those servers to figure out what was going on.

gnu parallel might be a nice choice for this sort of thing as well.

also for clusters i wrote a python script using libcloud that connects multiple terminals and does some bastardized map/reduce type helper ideas:

http://gist.github.com/281504

This tool looks awesome. For some of us, it's the first time at the running 20 EC2 instances at once rodeo, and a tool like this is helpful for testing and developing the parallelization and maintenance tasks that everyone else is being condescending about.

screen. worked well for 200 machines.

  #!/bin/bash
  if [[ "$1" = "-n" ]]; then
    DETACH=
    shift;
  else
    DETACH="screen -d -m"
  fi

  for i in $NODES; do
    echo $DETACH ssh $i $@
    $DETACH ssh $i $@
  done;

Reminds me of Tentakel. http://sourceforge.net/projects/tentakel/files/tentakel/tent...

Came in useful from time-to-time.

I use csshx, and I highly recommend it. One annoyance is that on password entry screens it occasionally does not show the password prompt. But generally speaking it's a great utility.

You should be using key authentication anyway so that shouldn't be a problem :-)

I've been using this tool for about a year and absolutely love it. It really comes in handy managing messes of EC2 instances.

I use pdsh for this sort of thing.

This is nice as a proof-of-concept. Now it needs to become a patched version of Gnu screen.

Does this mean that doing rm -Rf as root mean that one is technically "clusterfucked"?

i do something like this with multiple shell buffers and an emacs macro that pastes what i just wrote and hits enter in every buffer.

good stuff. can it also compare the response of each and report which one of these is not like the others?

iTerm2 has this feature with a much nicer layout.

http://www.iterm2.com

Where is this in iTerm2?

I would assume he's referring to "Send Input To All Tabs" in the Shell menu.

Ah. I've never seen that before. Nice to know. But I'm not sure this is a "nicer layout". And I'm an iTerm2 heavy user.

Why not just use dsh for this?

iterm's "send input to all tabs" does this rather well.

April fools?

Thought so too - until I looked at the Update history...

Well, ClusterSSH (a version that runs on Linux and uses xterm instances) is one of my everyday administration tools.

It's under 15 identical machines, so it is more trouble than it is worth to set up Puppet or something of that sort.

Oh my!

I use fabric http://fabfile.org/ which is insanely easy to use.

I use it even if I have one machine to deploy to. Better keep them deployments repeatable ...

Still, that makes me write a bit of Python. Not going to do that for a simple symlink change or one-off task of similar complexity.

You can actually do that with Fabric, too.

  fab -H system1,system2,system3 -- uname -a

Yes... April Fools.

Reply on news.ycombinator.com