SSH tunnelling for fun and profit: AutoSSH | Hacker News Reader

SSH tunnelling for fun and profit: AutoSSH

everythingcli.org

141 pointseverythingcli70 comments

I use sshuttle (https://github.com/apenwarr/sshuttle). It is super flexible and works as a poor man's VPN.

It also has much better performance when used over a flaky connection because it doesn't do TCP-over-TCP.

It doesn't really create a persistent tunnel like this thingy, but it seems worth mentioning to people who might find autossh interesting.

Sshuttle is awesome. Comparing the ease of use to openvpn, it's night and day..

Indeed I was looking to create a VPN and found it complicated - saw sshuttle and found it awesomely easy (blog post about it http://alicious.com/cheap-vpn-using-sshuttle/ [has aff links in post]). My commandline was:

    sshuttle –dns -rv root@128.199.xxx.xxx 0/0 –exclude 192.168.0.0/9 # start sshuttle

-dns routes dns requests over the link

-rv remote server details (v for verbose output)

-exclude to not send traffic from local addresses over the link (not sure if this is required)

I've found performance (bandwidth) on sshuttle to be poorer than OpenVPN.

It's definitely below what you get with a true VPN, but it's a great option when combined with an SSH server listening on port 443 to get any kind of "VPN" on a locked down network (like most airport WiFi, for example).

I wrote a guide on using Softener VPN which I found much easier to use than other VPN software:

http://www.selectedintelligence.com/post/128701492804/softet...

Our prototype used autossh actually but later we needed something a bit easier to manage. Disclaimer: I am not a networking or security expert.

Alternately, "ssh -D 1080 user@host" will give you a SOCKS proxy on 127.0.0.1:1080 that forwards all connections out of the remote host. No download required.

Yes this works fine for most cases. Although I've had DNS-related problems I've never been able to resolve properly.

Can you expand on that? I've never seen any problems with -D

Well the DNS queries aren't routed through the proxy.

Possibly because DNS queries are trying to be UDP?

Yes, though a plain SSH connection is likely to close periodically. AutoSSH is meant to prevent that.

this is easily solved by wrapping it in a "while true;". Still no download required.

That does TCP-over-TCP, though, which provides worse performance.

I use this every day and the performance is not that bad. You wouldn't notice the difference when browsing the web.

No it doesn't. SOCKS is connection- (stream) based, not packet-based.

Actually, that looks like a tool that could be used in conjuction with autossh. sshuttle provides data transport, autossh provides connection persistence. Thanks for the heads-up, this could be a useful tool for me :)

Too bad that Android blocks you from installing real Linux apps.

Not if you are rooted. If you want to get really crazy you can even boot up a whole Debian image.

For a persistent connection to a server, I like mosh (https://mosh.mit.edu). It works really well for roaming across networks and for intermittent or laggy connectivity.

Yea this looks like a terribly convoluted solution for something mosh has solved quite elegantly.

1. Mosh is about a decade younger than this ("why would anyone write this, when there would be a better solution in ten years?!" - Mosh was a response to the shortcomings of available programs, namely the combination of autossh+screen), 2. Mosh solves a slightly different problem (persistent terminal emulation across variable and unstable connection X persistent SSH connection - one has intelligent local echo, the other has tunnels)

I used autossh for 2 years, several times a week, and it is absolutely robust (like never failed, ever) when it comes to restarting a broken ssh connection

One thing the article glossed over is the SSH key. I set this up for remote access to a computer behind a NAT router that does not allow any incoming connections at all. For that to work robustly in case of a reboot, the private ssh key used by that computer must have no passphrase. That always makes me nervous, I ended up setting up the public computer side of this with a special user account that was as locked down as I could make it. All the ssh tunnel will do is forward port 2200 on the public machine to my computer's ssh port. Works great!

Or you could use a SSH agent (Pageant, ssh-agent): you enter your key passphrase once when starting the agent, and it handles the key auth on your behalf.

Why is this getting downvoted? It's clearly the right thing to do.

How do I enter the passphrase when the computer behind the firewall reboots?

Ah. Now that's a problem I have no solution for - only the same workaround as you describe (you have taken one more step than I did by having a dedicated host for the SSH connection, instead of just an account - with today's virtualization, this is easier than ever :)).

A challenge I've had with Autossh (and OpenSSH for that matter) is in getting the listen port during remote (reverse) tunnels when it is dynamically assigned via '-R 0:localhost:$SOMEPORT' where $SOMEPORT is on the machine originating the reverse tunnel, and 0 is a dynamically assigned open port on the relay/remote machine. It doesn't look like there is a good way[1] to get the listen port from the originating side (remotehost:$DYNPORT) without inspecting ssh stdout (complicated when using Autossh via systemd). Any ideas? On the remote side it is easy to find the listen port provided you have sudo access, but seemingly impossible without [2]. On the local side I probably just need to pipe stdout to a regex to find the right string and then redirect the captured port number to a file or named pipe, but it seems like it ought to be easier to get dynamic port assigned on the remote side from the local end of the tunnel.

1. http://www.nico.schottelius.org/blog/openssh-6.2-add-callbac...

2. https://gist.github.com/tomkinsc/c2932097796155f4fd94

Assuming autossh is going just rerun whatever binary ssh points to, then you likely wont need a separate binary for this. Just run

  while true; do ...; done

and be done with it. The author does say that

... Some scripts use a while loop, others encourage you to run a remote command (such as tail) to make sure you don’t run into timeout and various others. But actually, you don’t want to re-invent the wheel and stick to bullet-proof already existing solutions.

but the only case where that makes sense, and where a simple while loop will fail you, is if the ssh command hangs. Only then will this start making sense.

I'd put a `sleep 5s` or so in there as well, in case your wifi is off or something, and it's looping "no route to host" over and over.

...and a condition for this, and a bit of checking for that, and a SSH option, and a signal handler - and all of a sudden, you have replicated autossh 0.7, except buggier and in bash.

Replace bash with zsh and you have my life story.

1. SSH hanging is not altogether uncommon, esp. on repeated reconnect over flaky connections and/or crappy-but-stateful captive portals, experienced both; also, not every SSH server is OpenSSH, esp. various embedded dropbears have been imbued with nasty device-specific quirks. 2. a simple loop might fail you unless you've remembered to use -oBatchMode=true; also it would work iff every SSH connection is terminated cleanly; also also about 20 other minor warts.

Very quickly, the "simple loop" starts being decidedly unsimple, and you'll find yourself reinventing a square wheel. Been there, done that. Then I found that someone else has already been through the same issues and made a comprehensive tool for it: autossh.

No. SSH connections break when I close my laptop. Probably happened several hundred times by now. The ssh executable terminates in this scenario exactly never. I have to close and reopen the terminal window - can't even Ctrl-C. Come to think if it I've never once experienced the ssh executable actually exiting by itself. I don't know what problem this script solves, but it's not the ubiquitous experience of "When I close my Mac my SSH sessions in Terminal become frozen."

In the example of the author creating a reverse tunnel for mysql, I can't decide if I love or hate these types of tunnels. On the one hand it's nice to allow SSH to do all of the encryption heavy lifting, and to punch a single hole through the firewall. It's also nice that somebody sniffing around from the outside doesn't really have any idea what services are running on the box.

On the other handle it comes off as much too hacky for something I'd trust my production services with. It also involves two levels of authentication (PKI + whatever is built-in to the system).

Nice tool but not quite as reliable as I'd like it to be, especially on the reconnection-in-the-face-of-unreliable-transport front. I tried using it for my mailtunnel but eventually switched to a VPN over UDP.

you might take a look at zerotier.com. They provide tools/services to ease tun/tap much like docker eases lxc/containers.

For now openvpn seems to fill my needs but thank you for the reference, if I can't get this to work reliably over the longer term I will look at zerotier.

If you're running systemd you can use this:

    [Unit]
    Description=SSH remote forwarding to here
    After=network-online.target
    Requires=network-online.target

    [Service]
    User=autossh
    ExecStart=/bin/ssh -N -o PasswordAuthentication=no -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -R 2222:localhost:22 -o TCPKeepAlive=yes -p 9196 -i /home/autossh/.ssh/autossh autossh@example.com
    Restart=on-failure
    RestartSec=1200

    [Install]
    WantedBy=multi-user.target

This seems like pretty simple code. Could we not roll this into the official ssh client as a patch, but instead of port forwarding, use the existing keep-alive code and ssh signaling channel to check the connection and restart the session? Not only would this be cleaner (no need for wrappers), but we could use it on servers where port forwarding is disabled.

I understand the fun part but somehow missed the profit aspect

Can't remember how many times a persistent autossh reverse tunnel to a jump host has saved me from several hours of a field trip. A few man-months, I'd guess - that's quite enough profit for me ;)

Nice tool, but doesn't play well with ssh_config ControlPersist, which is essential to my use cases :-(

you could still append this option: `autossh -oControlPersist=yes|no"`

And then losing the advantage of ControlPersist; the ideal would be to have an always-on ControlMaster, but that would also be restarted once ServerAliveInterval*ServerAliveCountMax expires.

This is pretty slick.

This tool makes me afraid.

Anyone that can successfully use AutoSSH is not monitoring their private key usage. You should always need to approve each time your SSH key is used; whether via typing in your password, swiping your smartcard, or pressing the button on your yubikey.

I have a key generated for AutoSSH only.

Also you can limit to which hosts/ports it can connect with:

     no-agent-forwarding,no-X11-forwarding,command="read a; exit",permitopen="host:port" ssh-ed25519 AAAA

Pretty nifty.

On the server side, right?

Yes, in the authorized_keys

I've gone so far as to create a separate autossh user with separate key, with no shell access (by setting the user's shell to /bin/false). It still permits tunneling (call ssh with -N), but does not allow shell sessions. Perfect for when you want to access remote systems via a remote tunnel but do not want to give shell access on the relaying machine to the tunnel origin. It trusts that the machine originating the tunnel will not initiate a reflection attack (by making a local forward to the remote port itself on the originating machine and causing some log messages to appear until all file descriptors are in use)[1], but that's not so unreasonable.

1. http://www.semicomplete.com/articles/ssh-security/

Same here actually, the user has /bin/false as shell.

This is not how it is meant. When you use AutoSSH/SSH and need to enter a password for your key anytime you do a connection (which probably happens every 10min as a system engineer/admin/etc) it is probably useless.

I have many different SSH keys, each with different long passwords.

What u are looking for is `ssh-agent` which can take care of all the password handling.

Also unattended rsync (over ssh) backups rely on no user interaction at all.

On top of that, you can always use a separate user for the tunnel that doesn't have a login shell. Obviously still a problem if someone gets the private keys but not nearly as much so, as they'll basically be limited to tunneling/forwarding.

@Tiksi:

Couldn't reply to your thread directly, so here:

It shouldn't be a problem when someone steels your key. At least not when it is protected with a strong password.

As I said, you can and should always use `ssh-agent` and protect all of your keys with a password.

Now that it was mentioned, I am really not sure, if `ssh-agent` also works for processes started by `cron`.

The tool does nothing more in that sense than SSH itself. If you have an unprotected key it doesn't matter if you use it with just core SSH tools, your own scripts to restart things upon connection difficulty, or tools like AutoSSH.

Sometimes unprotected keys are unavoidable for automated admin functions - in this case you have to be careful what access is granted to logins via those keys, and be extra careful to secure the keys themselves including (if you don't generally) having a revocation procedure in place (even if that is as simple as a list of locations the key is accepted to you can remove it from them all) so you can easily deal with one being compromised.

You should always need to approve each time your SSH key is used

Why? I don't need to retype my password for every line of shell executed.

Please don't give "them" ideas, thank you.

Interesting. Does ssh-agent(1) also make you afraid? http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/...

The default settings are bad for the majority of sysadmins. See http://unixwiz.net/techtips/ssh-agent-forwarding.html#sec

With the ssh-agent defaults, if I have root/sudo access on a server you are logged into with ssh on, I practically have free use of your private SSH key.

I use gpg's ssh-agent emulation (See the man page for gpg-agent under --enable-ssh-support), and store my ssh private key on my yubikey. With this setup, the yubikey's light flashes when something wants to use my private key, and you can just touch the button on it to approve.

> If a root user is able to convince his ssh client to use another user's agent, root can impersonate that user on any remote system which authorizes the victim user's public key.

Yes, root can also simply `su - user` and then do all stuff you do.

But you should also not put YOUR private keys on a system that has a root user who should not have access YOUR keys.

(copying from https://news.ycombinator.com/item?id=10943166)

> The problem with ssh-agent defaults is that someone with root on the server you're SSH'd into can use your SSH private key (which is on your local machine).

> ssh-agent is like temporarily uploading your private key to each remote server logged into.

That is very different (and much more common) than an attacker having root on your local machine.

That is bad. Even worse was a default that the OpenSSH developers quietly added and folks have become addicted to. It also enables me to phish any company and get access to all their servers, bypassing 2FA with almost no logging.

Some time read up on SSH Multiplexing. The default is MaxSessions 10. You auth once, my phishing exercise can utilize your session to connect anywhere you have connected. Nothing will show up in syslog. I can take care of `lastlog` entries since folks cache sudoer privs for so long; or in some cases, don't even require a pw.

I will set up a github with a working example of how to set up a backdoor leveraging ssh multiplexing. I got the idea from a coworker (whitehat) that used a simple ruby script to completely pwn my system. The cool part is, it doesn't require exploiting anything beyond the helpful developer or sysadmin.

Does that boil down to "if root is malicious, Bad Things can happen to other users"? Well...duh. Why not keylog all terminals instead of fumbling with ssh-agent and connection multiplexing?

No. The problem with ssh-agent defaults is that someone with root on the server you're SSH'd into can use your SSH private key (which is on your local machine).

ssh-agent is like temporarily uploading your private key to each remote server logged into.

Do you have any material I can read up on to this?

In the same way, a helicopter is like a hot-air baloon. Except it's mostly not - the comparison is very much misleading.

The GP is mixing together SSH agent with agent forwarding, and private keys with key signing by agent.

There is last week's CVE-2016-0777, which is a vulnerability that enables the exploit you are describing, but in a properly patched configuration, without agent forwarding (nb: this feature is off by default), this is not the case. Same with connection multiplexing: off by default (for good reasons). In other words, either a) please explain how that would work, or b) please stop spreading FUD.

An illustrated explanation how ssh-agent is not like copying private key anywhere. http://www.unixwiz.net/techtips/ssh-agent-forwarding.html

ssh multiplexing is ON by default. The default is `MaxSessions 10`. Unless you have `MaxSessions 1` in your config, then you have it enabled in your environment. If you have 2FA and SOC1/SOC2 controls, then technically you are out of compliance.

Reply on news.ycombinator.com