Who's your SSH buddy? | Hacker News Reader

Who's your SSH buddy?

blog.jgc.org

149 pointsjgrahamc102 comments

This is a perfect example of what not to do: the guy has ignored the lessons of denyhosts (etc.) and came up with his own regular expression that doesn't do the right thing.

Code re-use please!

Or:

apt-get install fail2ban

Works out of the box to prevent brute force attacks against ssh, and can also be configured for other services, like web authentication, POP, IMAP, etc.

Just a friendly tip: Try to verify whether Fail2ban actually works and keep monitoring your log files (Logwatch). I once found some dumb ass trying to brute force his way into my Postfix server, unfortunately Fail2ban didn't catch it and I only noticed when Logwatch showed me the authentication failures (several hours and thousands of attempts later).

Ouch. Did you tell upstream so that they could fix this for everyone?

> someone I could call and give credentials to so they could log in and shutdown the machine.

I'm not going to be able to read out my private key over the phone. I guess this is only for password based authentication.

Well, ideally you'd work all this out with your friend before the emergency...

And this means you can add their public key as well, meaning you can control (i.e. revoke) their access easily and don't have to give them any root passwords.

Maybe a play off of In case of emergency, break glass with hammer and copy private key: a one-time-only retrieval of a private key at some URL, so when your buddy retrieves it, it triggers an email that goes out to you.

I think I need that for myself: still scared I'll forget a password or passkey somewhere, or forget what that one really long password is for.

Me too. But if it has a password on it, you'll be protecting your password with another password. If it's just an obfuscated, unprotected URL, then you're going to have to store the obfuscated URL somewhere safe.

I think this only works if you want to give credentials once to someone while still knowing when those credentials are being given, unfortunately :(

I recently began trusting KeyPass with my credentials.

It uses two mechanisms to protect my details:

- a secret key file (backed up on an usb stick and on dropbox). If you steal my keypass db you need this file as well

- a ridiculously long sentence that I can never forget for various reasons as a passphrase. It's plain text, so probably everything in there is in a dictionary, but - good luck trying to figure this out.

With this set up I'm relaxing now. I know that I ~cannot~ lose access or forget my passwords :)

Privnote (https://privnote.com/) can do exactly that. For added bonus, the message is stored encrypted on their servers and decrypted with a key in the URL's fragment.

"We store the notes encrypted in a way that only the sender and receiver know how to decrypt, which means we cannot read the notes sent through Privnote."

Sigh.

An alternative: give out a public key, but restrict it to run e.g. reboot only. Put something like the following in your ~/.ssh/authorized_keys:

    command="/sbin/reboot",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-rsa AAAAB3Nza...LiPk== user@example.net

See sshd(8) "AUTHORIZED_KEYS FILE FORMAT" for details.

I think you mean give out a private key.

No, he actually means get a public key.

You should never be giving out private keys, people need to generate those themself!

Why not just pastebin your private key, shortURL it and when the time comes you only have to give the shortURL.

A key alone, with no idea what it will unlock, is going to be a useless thing for anyone else.

Obviously I'm aware how crazy a public store of private keys sounds, but keys are really only useful if you know what lock they fit.

Just for the record, Pastebin is not completely secure. Even private pastes can still be accessed if you know the URL - they aren't password protected like your account may be.

But like you said, if you posted the key alone anonymously, it will be hard to guess what lock it fits.

> Just for the record, Pastebin is not completely secure.

Note that it already happens that people put their private keys into pastebin, by accident or probably by sheer incompetence.

> if you posted the key alone anonymously, it will be hard to guess what lock it fits.

As an attacker, I would simply collect as many of those "published" private keys as I can get. Then, when attacking a bunch of systems, I'd simply try one key after another on each system.

The important difference to the "real world" is that an attacker can try lots of "locks" at once (i.e. can connect to multiple target systems at once).

When attacking a bunch of systems, I'd simply try one key after another on each system.

This was a good idea in 1990. But now you get IP-banned after three bad login attempts, so you have to be smarter.

Good point. However, that's what botnets are for. Why "wasting" those for distributed denial of service if you could break in instead?

Sharing your private key for 'more security'? I will never share my private key with anything or anyone (pastebin or my ssh buddy).

My SSH buddy is iSSH from the App Store. Always available when I need him.

Assuming you've got a data connection, that is.

This is a pretty pointless idea. If someone has root on your machine for even 500ms that's already too late to trust any of its state without a complete reinstall (and even this is not enough for an advanced attacker).

Inevitably, if you have such a buddy, all they can really say is "yup, I'm not the only user here", and you're back at square one.

A better investment of the time would be systematizing your configurations, so in this situation you can just rebuild a new machine and kill the old one (ideally, after figuring out how the password was compromised in the first place).

Presumably the point is not "have someone that can fix your machine if it actually gets rooted", but "have someone you can call in case of an emergency who is competent".

Seems like for a business, this would be a great service to have available from the likes of Linode, which pays people to be there in any event, rather than your 'buddy', who might go on vacation when you do.

> This is a pretty pointless idea. If someone has root on your machine for even 500ms that's already too late to trust any of its state without a complete reinstall (and even this is not enough for an advanced attacker).

As the article says, he just wanted it shut down. Fixing it can come later, but stopping it from sitting on your network sending out millions of spam e-mails is still useful.

My custom monitoring program (probably not as advanced, it only does some light monitoring) accepts a text message back to shutdown the machine.

Not perfect but can work well as a line of defence.

That's cool. How does it work? Is the text message just via email, and the server can accept an email back with a specific command?

Servers are with Gandi - which has a hosting API. So... SMS gateway via some shared hosting I have had for ages. This calls a script which figures out which server I mean and uses the API to shut it down.

I did used to use Twilio until they dropped international text messaging etc.

What are you using instead of Twilio now?

A shared hosting account which has a free (5 credits a month) SMS gateway included - I think it is reselling txtlocal.co.uk (who look quite good, but I haven't gotten round to trying them out directly yet).

It was kinda thrown together in a few minutes after Twilio flaked :)

I'm not an expert in this things, but can't sms messages be spoofed quite easily?

I presumed they can be :) so my systems needs a text like this:

It is on my todo list to improve this at some point - but not had chance yet.

I have a similar system, but instead of a plain passcode, I have a (simple, and probably not very secure) hashing method that I can do in my head.

My message looks like this: <machine> <hashed passcode> <number>

The passcode is "hashed" with the number. When the server receives the message, it adds the number to a deny-list. That way, the number is only good for one use.

Also, the number has to conform to a certain pattern. I might change that to a pre-generated list of valid numbers, but for the moment, I can work out a valid number with pen and paper.

Not perfect, but I think it's good enough.

> But it made me realize that I need to tighten up my SSH buddy plans for the next time.

I'm going to ask a really audacious question: Why?

His systems all worked as expected. Injection attacks were thwarted and login attempts failed. He received (as it turns out, erroneous) notification about the login attempts and knows he needs to do something about it for the future.

Why is that something the sharing of credentials? Why is it that people still allow for remote root login? Why do people still allow user SSH access via password?

There's a better way, and it leaves you a damn sight better prepared for intrusion attempts than receiving SMS messages that, as he so perfectly demonstrated, were not actionable. Limit logons to PKI-only. Live happier. Sleep easier.

Though the main problem if someone breaks into your system, the first person you'd suspect is your SSH buddy, because there is a possibility of them using a public machine to log in. Who would want to carry the burden of being your first suspect?

Something to sincerely rethink is the idea of shutting the system down in the first place. Call the network provider & have them unplug the ethernet.

Turning the system off could potentially reduce your ability to audit where they originally got in. Anywhom.

SSH keys :)

Many messages here like "why not do X?" or "why in the world are you doing Y?" All good suggestions, but sometimes X or Y may not be good enough, or you may have implemented them incorrectly (like the author's errant regex), or the completely unexpected Z might happen; no one expects Z.

I don't see anything wrong with having a human backup. Heck, what if you die and things need to be wound down?

Dying is outside the scope of application. Please contact nearest heavenly body for ticket submission process.

i don't get this. is it about a personal site? if so, how paranoid can you be? or is it about a professional site? if so, who not try calling a colleague?

At the risk of talking about the actual topic at hand instead of diving off into minutia about putting SSH on your phone, I'm not sure what this gets you, from a paranoia point of view. Assuming the idea is that you want 100% coverage of your server, this might marginally increase your odds, but not really all that much. Times when you aren't available to administer your server are likely to correlate highly with the times when your buddy isn't available. And what if you'd never gotten the SMS in the first place because you were entirely out of service?

One person simply can't cover a server 100%, adding a "buddy" doesn't help much. Either you need a full-on netops operation, or you need to be able to deal with not having 100%.

Umm... my iPhone, iPad and MacBook are my SSH buddies?

Why not have the IDS shut down the machine itself?

Isn't the real problem that he only has 1 sys-admin for his servers (as far as I can tell)? I mean he can have as many SSH Buddies as he wants but if the alert comes in when he is asleep or drunk or whatever it's pointless.

Maybe the alert goes to multiple buddies who have the tech skills to handle it ... maybe he could be their SSH Buddy in return? In other words, rather than a SSH-Buddy, have a pool of server sysadmin friends for emergencies.

ps. If the original poster is reading this, I would have posted this on your blog but I didn't have other account passwords to hand ... if your moderating all comments anyway, why not allow anonymous comments?

I can't believe no one has suggested this yet: this is the PERFECT argument for two-factor authentication. He's even already sending texts to himself.

For every new connector, send some sort of code to your phone, and require it to be inputted back into the computer before any access is given whatsoever. Summarily disregard every single connection and command given without this code re-input.

You're even most of the way there: if you're sending messages to yourself, you might as well implement this behavior with it.

The unix on call operator in my data center is my ssh buddy

No ssh client still on iphones?

I should have had one installed. Now I do. But I still need an SSH buddy because I might not have data access (for example, if I am abroad).

Hook it up so that a few preset operations can be called via SMS, e.g. shutting down the server?

You know that SMSes are pretty insecure right?

I don't think it really needs to be secure for this purpose. It's easy enough to fake the number you're sending from, and probably (I've no idea) easier enough to listen in on SMS messages sent (assuming it is for the sake of worst-case).

Have the server set up to require a.) sent from correct number and b.) correct password following the "shutdown" command.

This isn't a system set up to be regularly used, it's about killing the server until you can get to a computer, when you're under attack. So have the server only accept the password once then disable the system - once he's back and made sure the server is fine, that code can be changed, so it doesn't matter who was listening in on the original message, after it was sent that code stops being useful.

It's also easy enough to have access to secure Internet to pretty much everywhere in the world.

Plus you don't really need that secure of an access. Any given access would do... plus a limited login that can only sudo shutdown.

It's expensive, but have data-roaming enabled on your phone. You'll only be using it for emergency server-maintenance after all - and surely you can bill it to your employer in those emergencies anyway.

If this is for an employer, there should probably be someone monitoring the servers when jgc is on vacation.

There are plenty of employers that only have a single sys-admin on staff.

I've had an "ssh buddy" for years, but for a variety of things such as asking for a "download on how to hotwire a motorcycle".

I have an agreement with many friends to be an operator so they can look something up on the web, or anything else requiring some efficient technical prowess.

We just call up each other and say "operator", and if you're near a computer, you help out with whatever it is.

Ultimately this turned out to be a false alarm. Although the machine was under attack (on many levels: there was activity hitting the packet filter, trying all sorts of injection at the Apache level and having a go at SSH) the actual alert (based on looking in auth.log) was a false alarm based on a bad regexp.

Any particular reason you didn't want to use Snort or Bro?

Alternative Solution, with iptables and perl below.

You send the magic string to any tcp port and it'll instantly kill all SSH logins and disable the root shell. Send the string again and the process is reversed. Caveat: if your syslog contains trigger entries more than a year old this will blow up.

I also recommend you disable root logins and password authentication, but if you insist on enabling them, this may work for you. Modify as necessary.

  iptables -t raw -A PREROUTING -p tcp -m string --algo bm --string "_-()ThisIsAReallyLongAndComplicatedRandomStringToMatchOn()-_" --from 0 --to lengthofthelongstring -j LOG --log-prefix "29CharacterMaxTriggerString "

  #!/usr/bin/perl
  # sentinel - disable root based on a syslog trigger
  # Copyright (C) 2011 Peter Willis <peterwwillis@yahoo.com>
  use strict;
  use POSIX qw(mktime);
  my $LOG = "/var/log/syslog";
  my $LOCKED = 0;
  my $TRIGGER = "29CharacterMaxTriggerString";
  my %M = ( "jan"=>0, "feb"=>1, "mar"=>2, "apr"=>3, "may"=>4, "jun"=>5, "jul"=>6, "aug"=>7, "sep"=>8, "oct"=>9, "nov"=>10, "dec"=>11 );
  
  for ( ;; ) {
      sleep(1);
      open(F, "<$LOG") || die "Error: $!";
      for ( ;; ) {
          sleep(1);
          while ( <F> ) {
              select(undef, undef, undef, 0.001);
              #print STDERR "Reading \"$_\"\n";
              if ( /^(\w+) (\d+) (\d+):(\d+):(\d+) \w+ kernel: $TRIGGER / ) {
                  my $time = time();
                  my $stamp = mktime($5, $4, $3, $2, $M{lc $1}, (localtime($time))[5]);
                  if ( $stamp > $^T ) {
                      #print STDERR "Found Trigger\n";
                      trigger();
                  } else {
                      #print STDERR "Error: found trigger but timestamp $stamp is before script begin time $^T\n";
                  }
              }
          }
      }
      close(F);
  }
  
  sub trigger {
      if ( $LOCKED ) {
          system("/usr/bin/chsh -s /bin/bash root");
          $LOCKED=0;
      } else {
          my @procs = map { @_=split(/\s+/,$_); $_[1] } grep(/^root\s+.*sshd:/, `ps -aux 2>/dev/null`);
          #print STDERR "Killing processes: @procs\n";
          kill(15, @procs);
          kill(9, @procs);
          system("/usr/bin/chsh -s /bin/false root");
          $LOCKED=1;
      }
  }

Wouldn't a OTP scheme like OPIE be able to do this decently? My own computers do not permit root login via ssh, but still it'd be more or less a trivial matter to set up a local account for your "SSH buddy", and then over the phone give him the next OTP in sequence for the root account or so for a successful su - -c halt.

Isn't there a service where you can SMS something like shutdown, restart or other commands for the box to respond to, that would help in that instance where net access is poor

Also why would you allow either password based and root login when you are "pretty careful with machine security"?

I had to smirk when I clicked on complex password scheme.

I found that password scheme pretty OCD and not very secure considering once you have that paper, you have a pretty good chance of very quickly brute-forcing your way in.

If someone breaks into your house or office to get that piece of paper, your problems are bigger than a compromised server.

But a piece of paper is easier to snatch than having your passwords written down nowhere - without stealing a whole wallet or breaking into houses.

If having your password on a note is good enough for Schneier ( http://www.schneier.com/blog/archives/2005/06/write_down_you... ), it's good enough for me. For pretty much the same reason. If you're going to even use a password, it should be a good one. And if it's a good one, you aren't very likely to be able to memorize it.

With upper and lower case and "1337" speak thrown in between and a few other characters replaced, it should not be that hard to come up with a decent enough password that you can even remember.

And why not a password manager instead of paper? Would be even better if it was one that reads your fingerprint or so.

I think that guy needs to work on his paranoia.

...or at least work on the security in a more well-thought way!

can you not ssh from your phone?

Any idea of what he uses to send SMS from his box?

If you don't want/can afford an sms-gateway, you might want to look at notifo [1]

[1] http://notifo.com/

Twilio would work.

http://www.twilio.com/docs/api/rest/sending-sms

I dunno about jgc, but I just send an email to myphonenumber@txt.att.net

Works like a charm!

Before you get into half-arsed solutions like having an ssh buddy, disable root login and password authentication.

I assume that he already did this. That the alert comes when someone assumes root. But maybe I'm being naïve.

I don't understand what's 'half-arsed' about deciding up front who I can trust to have login credentials for my personal machines.

Prevention is better than cure.

Preparation is better than being unprepared.

(Not that I'm arguing your point. Just taking all steps that you can think of to prevent something, doesn't mean you'll actually prevent it. Being prepared for a worst case is always a good idea.)

Ideally you should not have to circumvent the default PermitRootLogin settings; it is there for a reason. And no need to give out root passwords, if you are so security conscious.

Our polo-necked messiah has given us iPhones that can even run an SSH client so how sort-of pointless to talk about "ssh buddies" and showing iPhone screenshots in the very same article? Since you were getting IDS alarms on your iPhone anyway, why didn't you SSH in and shutdown the machine yourself?

And on the root login bit: you would only need a sort-of dummy account for you logging in or for your "ssh buddy", if you love that idea. And all they need to be able to run is "sudo shutdown".

I was waiting for the article to mention the fact that you can SSH from any capable smart phone. Then he/others are saying using SMS to send commands from his phone is a good idea. [facepalm]

Get a mobile ssh client, set it and your sshd up with RSA keys, setup port forwarding on your home router on a non-standard port, disable Root login and move along.

The SMS part was only added later... the proposed "solution" was: SSH buddy!

I don't know why this was linked here and got all those upvotes. I don't see anything special about it, really.

You may have a cell signal but no (or prohibitively expensive, in the case of traveling abroad) data access.

Then again, if one of my servers was being hacked, I think I'd be ok spending $50 in data roaming to shut it down.

Although, there's the case of a really slow data. On T-Mobile I'll occasionally get an EDGE signal too useless to even establish an ssh session, let alone be interactive enough to use ssh. A text message can often be much more reliable and easy to use. Not sure about the security implications related to SMS remote control, though.

Well on the surface nothing, however login credentials can be stolen easily. Focus is on "easily". Sure keys maybe hijacked but the problem becomes non trivial and I am all for making burglars jump through a monkey bar before they even get to the door.

Nothing wrong with deciding that. The proper way to give them access, however is to give them an SSH key, not a name/password set.

Doesn't seem like front page news. Use key based auth and disable root login. You can also get an ssh client on your iphone if you don't mind keeping your private keys on it, just make sure to use a long password and be prepared to change all your server keys if you lose your phone. I also ban every IP that even tries to login as root.

The remote root login was my first thought too. Especially when I saw the sentence "... I'm pretty careful with machine security ...".

What's bad about password auth if your password is strong enough?

With passwords you are handing out what is needed to log in. With the ssh public key scheme, you only prove that you have the private key, but don't hand it out to the server.

I don't feel safe carrying a private key around with me on my mobile. It could be copied without my knowledge and brute forced in privacy. Whereas with password authentication, I can impose limits on the number of tries and also on the time between the tries. Am I wrong in thinking this?

Encrypt your private key!

This is very easy to do, and quite important for devices that can be stolen - including your desktop computer.

It's easy enough to setup ssh-agent on your desktop so you don't have to keep entering the password.

If you have a copy of the private key and the public key, you can bruteforce the passphrase. So use a big passphrase instead of a password.

This is why I have a completely separate private key that I generated for my phone, using a different password that's unrelated to the passwords I use for all my other machines' private keys, and I use password protection on the phone itself to thwart casual intrusion. If I lose my phone, it's trivial to remove the phone's public key from my servers' authorized_keys files, at which point the phone can no longer be used to log in.

passwords are replayable tokens. one-time passwords are not. people who actually are careful with machine security use ssh keys + sudo/opie.

In /etc/ssh/sshd_config do:

PermitRootLogin without-password

That way root must use a public key to login, but other people can still use passwords.

(But watch out for sudo.)

Yes, but the issue here is not root login, it's the prospect of something going wrong with your server(s) and having a trusted buddy around whom you can call on for assistance. It's actually pretty reasonable. My servers always crash when I'm literally somewhere in the jungle in Asia and can't possibly get to an online connection for at least a day. It's great if you can just call someone who takes care of the most pressing problems if/when they happen.

My servers always crash when I'm literally somewhere in the jungle in Asia and can't possibly get to an online connection for at least a day.

Get a watchdog timer.

I can't quite decide whether I really failed to bring my point across or if you're just pulling my leg here. Okay, I'll be more specific. Things happen to servers when you're not around. It helps to have someone you can call who can fix small things when they go wrong. You can't meaningfully automate every single thing that might happen. It might not even be a technical malfunction, maybe something unexpected needs to be done and it requires a person to do it...

I have a `poweroff' account on my machine with sudo poweroff as a shell with possibility to use the command without a password.

now that your friend logged as root, you did have a breach.

SSH buddy? I don't even have a porn buddy! http://www.youtube.com/watch?v=mgnDHbeVGG4

Reply on news.ycombinator.com