A visual guide to SSH tunnels | Hacker News Reader

A visual guide to SSH tunnels

robotmoon.com

963 pointsbrendanfalk80 comments

Great write up!

One thing to add is that you can even open tunnels during an interactive session without disconnection.

To do this, type the escape command sequence ~C (will not show) and it will drop you to the control prompt. You can then add tunnels.

     ssh>
     ssh> -L 8000:localhost:9000 
     Forwarding port.

I’ve been using ssh for 25+ years, I wrote a (in hindsight, pretty awful) article about it for Linux Journal, and I never knew this.

Is ~ a literal ~ character? Or notation for some modifier key? I only knew about ^ to mean "control".

Literal ~. Its input as a sequencenot as a modifier. Iirc the sequence ~h or ~? prints a little help page

Warning, if you read comradesmith's comment even more literally and type "~." (tilde and period) in an SSH session, that will close the connection and exit the client. It is useful when a remote application hangs and does not let you exit gracefully: https://apple.stackexchange.com/questions/35524/what-can-i-d...

Yes, but you need to press enter before using it. Try typing `⏎~?' in an SSH session.

Well, there's something I didn't know. Thank you for the time and tab savings!

Since this tip is popular I'll share another related tip.

When using nested ssh session, you can send escape commands to any of them by the number of tildes. For example, if you ssh into Server A and then from Server A to Server B:

    ~~.   # drop session from Server A to Server B
    ~.    # drop original session into Server A

I sometimes have to hit a <return> before the ~. disconnect works. Not sure why.

Because iirc, ssh only interprets the tilde if it's immediately preceded by a newline.

There's a really fancy new -R feature that I love (added in 2017 I think).

ssh <target> -R <just a port number>

This opens up a localhost port on the target that acts as a socks server which tunnels all your traffic through the source machine.

This is great for machines that you can SSH into, but are otherwise completely isolated from the network, or are monitored heavily. You can jump on, and pull down everything you need from the existing SSH connection, rather than using the machine to make requests out to the internet directly.

This is also good if the source machine is a web server or something on a secured network which can SSH out, but not much else, and the destination is your command and control server on the internet. Then it opens up a socks port on the C&C machine that gives full access to the internal network, impersonating the source machine.

Every other -R is a point to point TCP connection, but setting up a SOCKS proxy with -R is magic. More analogous to a reverse -D than a reverse -L. Super useful.

> (added in 2017 I think)

Indeed: committed in September 2017, released in v7.6 in October 2017.

This[1] is probably the best, easiest and most (visually) effective way to learn about SSH tunnels.

[1] : https://unix.stackexchange.com/a/118650/289353

At least this one is actually visual. Perhaps the author meant "A written guide to SSH tunnels"?

This is the one I always come back to when I'm a bit confused about the ordering of ips and ports.

definitely more visual than this article...

Drawing some diagrams helped me understand remote port forwarding and OpenSSH's port:host:port syntax: http://dirk-loss.de/ssh-port-forwarding.htm

I think it's way clearer!

Thanks! I pinned your page on my evernote.. I always get confuse everytime I need to make a ssh port forward. After a while I get it but I spend time (and some bad words :P)

One common use is to secure vnc, which has no crypto built into it.

On the remote host, have the vnc server listen only on localhost:5901

This assumes everyone who can ssh to the server should be allowed access.

On your workstation, form an ssh tunnel to that remote host and its port, and link it your own localhost:5901

Open your vnc client GUI and set it to open 127.0.0.1:5901, voila, vnc session.

More fun things: say for example you have a big remote xen dom0 with a lot of unique qemu HVM VMs running on it. Configure each of their .cfg files for xen to spawn a vnc server on localhost only and a unique port number per domU (5902,5903,5904,etc). Then use the same method to connect.

I am always amazed at what is possible with the linux command line, and it‘s something one can really fall in love with.

Note that it still exposes the connection to all local processes, on both hosts (except network-restricted services or with non-typical host-local firewall rules), though it should be fine, if you're using semi-reasonable passwords.

SSH supports unix domain socket forwarding and even cross unix/ip forwarding with a syntax like:

  ssh -L /path/to/sock:/path/to/sock
  ssh -L /path/to/sock:localhost:5901
  ssh -L 5091:/path/to/sock

Then you can have user-authenticated host and transit secure password-less VNC connections.

Unfortunately I haven't been able to convince tigervnc viewer to connect to a unix domain socket, so I don't have a fully scripted connection, but remmina and tigervnc server work fine.

Also, forwardings / connections fail if the socket already exists when the listening side starts up (often an unclosed leftover), so I rm them before starting the ssh connection / vnc server and it works reliably.

Wow that's amazing, I've been cobbling together socat processes because I didn't know this was possible.

I do this for a number of hosts simultaneously with macos and map them specifically in my .ssh/config:

  Host mac-a
    ...
    LocalForward 5903 localhost:5900

  Host mac-b
    ...
    LocalForward 5904 localhost:5900

then I open the directory /System/Library/CoreServices in the Finder and drag Screen Sharing.app to the dock.

to remote into machine mac-b, in a terminal window:

  ssh mac-b

The first time, click on "screen sharing" in the dock, and in the connect dialog enter localhost:5904

After that, you can right-click on screen-sharing in the dock - even when it's not running - and you will see a context menu item for "mac-b 5904.vncloc"

By the way, mac to mac screen sharing is very well done (will they axe it next?)

- it is very fast

- the keyboard stuff just works

- you can copy/paste even rich text between the local and remote macs

- you can drag and drop files from the remote desktop to the local one

These are simply amazing. There's a stack overflow answer similar to this which I found very intuitive to understand https://unix.stackexchange.com/a/115906/254044

SSH tunnels are my favorite tool for NAT-busting. I always have to look up the cryptic syntax, but with one strategically placed Raspberry Pi, you can basically get from anywhere to anywhere.

And it works just as well on a locked down corporate network as it does on a home network. Why people put up with garbage VPN software is beyond me.

SSH isn’t a replacement for VPNs. I’d argue that they complement each other.

For example, SSH port forwarding is TCP only, so if you need UDP, you‘ll need a VPN.

[note I am wrong here - it doesn't support UDP - see below]

The ssh SOCKS proxy supports UDP. I use Firefox with a SOCKS proxy via ssh (e.g. ssh -D 2222 hostname) to get around geoblocking and accessing papers via an institutional account. The tsocks program can also let you use a SOCKS proxy for an arbitrary command in linux. I find it much more convenient than a VPN, as it can be easily applied to single programs.

Are you about this? Last time I've checked SSH SOCKS5 only supports TCP traffic, UDP isn't supported and needs extra steps through tools like socat to wrap UDP in TCP.

http://lists.mindrot.org/pipermail/openssh-unix-dev/2017-Jan...

It looks like I was wrong about UDP being supported by ssh. Sorry about misleading everyone. I've verified with nc that it doesn't actually forward UDP. I'm amazed that streaming media over this seems to work fine on most sites, though.

Many applications these days can fall back to TCP because UDP is often blocked (almost never for good reasons, but that's another discussion). For real-time applications, that fallback usually comes with a loss in quality/interactivity, though.

Alternatively, you might be unknowingly using un-proxied UDP: Many AV systems only enforce IP restrictions on the signalling protocol (which is often TCP); the actual media can then take whatever path it wants.

By the way, it seems like you can forward UDP over sshuttle [1] when used with tproxy (which seems to be a Linux-only feature)!

[1] https://sshuttle.readthedocs.io/en/stable/tproxy.html

For single applications that are able to "cooperate" with being proxied, a proxy server can make much more sense than a VPN, yes.

The less cooperative your applications are (even though cooperation can be somewhat enforced using tsocks, as you mention, which can be further made to apply system-wide using sshuttle [1]), the more a VPN will start to make sense.

[1] https://github.com/sshuttle/sshuttle

Note that the default port for SOCKS(5) is 1080, and the de facto standard for (forward) web proxies (e.g., Squid) is 3128, so one may want to make of habit of using one of those ports.

3128 is a common http/s proxy sure, but so is 8080, and I've seen 80 a few times too.

1080 is often default for socks, but I've also seen places where it's 1085 (for socks5 -- 1084 for socks4)

Ultimately it can be whatever you want.

Indeed. For VPN, use Wireguard. It’s open source, easy to configure, and works excellently.

https://www.wireguard.com/

Actually, OpenSSH has built-in TUN/TAP support using -w<local-tun-number>:<remote-tun-number>. You can create a layer 3/point-to-point/ TUN tunnel or layer 2/ethernet/TAP tunnel.

I used to run a site-to-site VPN between two sites a couple of decades ago with a simple script doing just this. These days we'd use OpenVPN or strongSwan for such use-cases but despite the 'hackiness' of the former approach, it worked reliably for years.

OpenSSH VPNs suffer from TCP-over-TCP [1]. I'd only use them as a last resort.

Sometimes I actually wish SSH would offer UDP support for precisely this use case, but arguably, TCP VPN support is already a stretch in terms of scope for SSH.

[1] http://sites.inka.de/sites/bigred/devel/tcp-tcp.html

One can further tunnel SSH inside of TLS with Stunnel [0] to get through firewalls that block SSH but permit arbitrary TLS connections. Loading a website through this requires three layers of TCP but assuming the connection is good this should be okay.

A couple reasons for a regular VPN: they can be made automatic and persistent so the tunnel is always ready for use, they can tunnel other protocols besides TCP, they can tunnel entire networks, and using UDP as the tunnel protocol helps avoid the TCP-in-TCP congestion issues.

[0] https://en.wikipedia.org/wiki/Stunnel

Stunnel and ssh tunnels don't put TCP inside TCP so it shouldn't risk any TCP-in-TCP congestion issues?

Yes, and piecewise TCP is often even faster than point-to-point TCP, especially on flaky connections, or when the transient connections are on hosts with lower RTTs (outgoing gateway closer to destination). It will likely be faster than a UDP VPN.

It still does have overhead, so lower bandwidth. Also, TCP, SSH and/or TLS forwarding of UDP, IP or Ethernet can have issues with delays, re-transmits etc (TCP-in-TCP).

SSH is a TCP connecting though, isn't it? How do you avoid TCP inside TCP when both are TCP? Oram I missing something?

It just relays the packet payload (i.e the actual data bytes inside the TCP stream are read out from the source, then sent along to the destination inside a different TCP stream. TCP/IP headers aren't included or replayed).

Why don't people use unix domain sockets for local connections? They are pretty much the same thing as local IP connections, except access rights apply as on files, so you're not exposing connections to pretty much all processes.

Domain sockets can be forwarded via ssh with the same -L and -R arguments, including cross unix/ip forwardings.

last time I tried ssh does not delete the socket file when the connection closes and complains the file already exists on a new connection.

It's a relatively recent feature in openssh (in the past decade for upstream; not sure when it first landed in an Ubuntu LTS, which for practical purposes is the earliest one might start using it regularly)

For those interested in the topic of tunneling, I highly recommend The Cyber Plumber’s Handbook [1] to gain an even deeper understanding and examples beyond SSH.

[1] https://cph.opsdisk.com/

Appreciate the shout out magicconch! Just added a HN family discount to pick up the book, lab guide, and lab access for $10.

https://gum.co/cph_book_and_lab/hackernews20210207

very cool, just grabbed a copy.

Appreciate the support pixelface...hope it helps!

Just tried to grab a copy but gumroad returns a 404 after trying to pay with Paypal. Do you sell it through another shop, too?

Edit: Never mind, it seems it was a problem with Firefox. Chromium worked.

Let me know if it's still giving you trouble...contact info is in my profile.

Thanks for the HN discount! Purchased.

Very cool, I was unaware of this product. Thanks for the deal! :D

Is it just me or is this site a bit of a disaster on mobile?

Yes gave up it’s too awkward to read.

Yep, but it works well for me in landscape view.

Could read just fine on iPhone 12p after zooming out; even in portrait

On desktop firefox all of the code snippets are black-on-black so I have to select the code to read it.

Nice reference.

Reaching target hosts multiple jump-hosts away, and going the wrong way through multiple firewalls segregating those lans, is another use-case of tunnels I've found handy.

i.e.

your_remote_client_host->internet->fw1->lan1->fw2->lan2->fw3->target_host

Target_host can be reached by doing tunnel-in-tunnel-in-tunnel-in-tunnel. Each tunnel gets you past one firewall. The final tunnel you can just ssh to fw3 via a local tunnel and a local port on your_remote_client now takes you straight to target_host.

Nicely written!

I once also wrote a guide to access remote IoT devices using SSH tunnels: https://phillip.dornauer.cc/unix/iot/2019/04/12/set-up-ssh-t...

Absolutely brilliant! I've had to explain local/reverse/dynamic port forwarding to people enough times that I've written a text file I can just send them, but I can finally replace that with a link to this page.

Really nice and clear. One addition: the final example uses ProxyCommand, I find ProxyJump much more useful: you can specify multiple hops clearly and even specify different private keys for each hop.

ProxyJump indeed seems more useful, but has less support (it's a "newer" SSH feature). Do you know if it needs to be supported by the openssh libs of intermediate hosts, or just the initiating client?

As far as I remember, it‘s implemented on the client only. (It can be replicated with ProxyCommand and a few invocations of cat/nc on the proxy host, if I remember correctly.)

ProxyCommand is useful for proxying with a non-SSH protocol. For example when combined with proxytunnel or similar you can run your SSH connections over an HTTPS proxy, which appears as regular web traffic to the network operator.

That comes in very handy. Bookmarked.

I also found the following very nice: (Same info but presented in a different way.)

https://unix.stackexchange.com/questions/115897/whats-ssh-po...

https://unix.stackexchange.com/a/115906/9745 I’ve always shared this with my team. I think OP’s post is good, but prefer the visuals of this for the `-L` and `-R` features.

Great resource. Most tutorials on SSH tunnels are strangely incomplete. This one seems to cover all the important points, and does so in an approachable way.

Every once in a while I wonder how the SOCKS5 stuff works at the protocol level, I would like to be able to SSH out without running a proxy process locally, just talk to the ssh deamon on the relay server directly. The OpenSSH code is a bit dense but it looks like it creates dynamic tunnels that are torn down when the connection closes. Anyone confirm that is what happens behind the scenes?

Great, I can refer people (and myself) to this page !

A lot of tunneling tutorial, medium articles and blog post mixes local and remote port forwarding and use the words interchangeably which cause a lot of confusion.

There’s also -w which sets up a point to point tunnel which you can route over - even more powerful than the SOCKS server

In tmux, when opening multiple sessions to the same remote server, how do you avoid typing the password repeatedly?

Add your ssh pub key (id_rsa.pub) to authorized_ssh keys on the remote servers will automatically authenticate you.

https://www.ssh.com/ssh/copy-id

Look up ‘shh multiplexing’. This makes subsequent sessions share the same connection. I don't know a particularly good tutorial; maybe see https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing

Why not visualize them as actual tunnels?

Really nice guide! This stuff really helps getting into the topic!

I was hoping for a series of tubes.

Thanks for this amazing guide!

In a computer networks (the Internet), if I can see you, you can see me.

You might want to be a bit more specific than computer networks. Monitor/ promiscuous mode is a thing, and so is dropping packets. It's not much use outside of wifi anymore, but there's still a few dumb hubs around.

Reply on news.ycombinator.com