Making an SSH client the hard way
tailscale.com
tailscale.com
Would it be possible to bundle the same into a portable application allowing you to use Tailscale without installing it? My understanding is that currently if you can't install Tailscale on a client you need to use Subnet Router. https://tailscale.com/kb/1109/devices-without-tailscale/
There‘s an utility called SSHuttle that does something similar for SSH instead of Tailscale/Wireguard: It redirects all sockets usage to go through an SSH connection, to allow usage of SSH port forwarding without explicit SOCKS support on the app‘s side.
there's a rather large misunderstanding on its github page:
> You can't use openssh's PermitTunnel feature because it's disabled by default on openssh servers; plus it does TCP-over-TCP, which has terrible performance.
it doesn't do TCP over TCP, it's a bytestream over TCP (exactly the same as shuttle)
something like OpenVPN running in TCP mode would be TCP over TCP
That SSH feature is indeed used for packet forwarding over SSH using TUN/TAP, i.e. packet-over-TCP, and by extension TCP-over-TCP.
Tailscaled comes with a socks5 proxy
Yup. :)
In fact, that's mentioned in the original public bug: https://github.com/tailscale/tailscale/issues/3157
If, say, the adblock Chrome extension you're using gets bought by a malware operator and backdoored[0], now it also has SSH and VPN access.
[0]: https://www.wired.co.uk/article/fake-chrome-extensions-malwa...
The browser, opening connections from within the browser engine, doesn't have the keys for SSH or VPN access.
Installing almost anything in your browser is usually a matter of a couple of clicks.
This was true even before this new feature.
The new threat model is entirely psychological.
I'm not commenting on whether it's a good or bad idea, but we should at least be talking about the same thing.
There's not really a "safe" website when the browser is malicious.
Wow, and they're proud of it.
Did you experiment with the new WebTransport API [0] at all? It's only supported in Chromium browsers, but seems promising for this kind of use case.
If Tailscale's client was a userspace construct bound to a specific user SSH program, maybe fine. But Tailscale's client is a regular VPN client. What happens if you connect to the Tailscale VPN, open a malicious but sandboxed app of some sort, and that app connects to the target on TCP port 22.
For all that it's a seriously unfinished product, Cloudflare's SSH offering seems better thought out. Perhaps Tailscale should find a way to issue a short-lived certificate and use that in addition?
(It looks like regular sshd could almost be convinced to handle this. If the SSH_CONNECTION environment variable were passed to the AuthorizedPrincipalsCommand helper or if the source and destination were available as '%' tokens, then AuthorizedPrincipalsCommand could do the Tailscale tuple lookup and use it as a second factor in addition to a short-lived certificate (or regular SSH key or whatever). I bet openssh would accept a patch for this.)
No check mode reuses the auth of the tailscale client, check mode authenticates the ssh connection itself
I feel like there should be a lightweight way to use SSH certificates, so you could use it independently or on top of Tailscale. Like a server (CA), client and daemon on your machines that should be reachable via SSH that handles short lived certs and authentication of clients. But I'm not aware of anything like that.
I have used Teleport before but it seemed not that great for machines not publicly reachable, because then all the traffic goes through a proxy.
Gravitational’s Teleport seems pretty good, but it’s heavyweight and doesn’t have any pricing appropriate for small businesses.
The configuration is:
Host vm.example.com ProxyCommand bash -c '/usr/local/bin/cloudflared access ssh-gen --hostname %h; ssh -tt %r@cfpipe-vm.example.com >&2 <&1'
Host cfpipe-vm.example.com HostName vm.example.com ProxyCommand /usr/local/bin/cloudflared access ssh --hostname %h IdentityFile ~/.cloudflared/vm.example.com-cf_key CertificateFile ~/.cloudflared/vm.example.com-cf_key-cert.pub
Think about this a bit. Contemplate what happens if you use sftp, vscode remoting, or anything else nontrivial. Hint: “cloudflared access ssh-gen” is not actually any sort of proxy, and the ssh -tt command is a kludge that should, if openssh were more on the ball about inherited file descriptors, should not work at all.
The right way to to this is to use Match … exec. Or to ask openssh to add an option for a command to execute before reading IdentityFile. Or to ask for an IdentityFileCommand option. Or to use a custom ssh agent.
* garden-variety web attacks (i.e., XSS, CRSF, etc)
* attacks that might become viable against the browser (for example, Mobile Safari has a history of vulnerabilities)
* various attacks against the backend web server (API attacks)
* attacks against the WASM layer
* CDN injections
* Tailscale's backend (various types of injections, timing attacks, or deeper attacks on Tailscale's infrastructure like the nightmares of HeartBleed, Shellshock, Meltdown, etc)
Realistically, this essentially (actually, literally) opens a remote root shell into your entire infrastructure through a web page, with apparently nothing more than matching an IP address pair (https://news.ycombinator.com/item?id=33361837) to authenticate.
What could go wrong?
This design with its loose coupling between authenticated user and IP addresses for high-value targets makes me view Tailscale's security model in a whole new light.
Or, did you mean attacks on SSO? If that's the case, then SSH web wouldn't make any difference. Someone authenticating themselves could use regular SSH or whatever.
Similarly, Tailscale backend is already subject to the vectors you mentioned (API, side-channel attacks). This feature doesn't add any new attack vectors.
Again, attacks on browser means end of game already. Someone can use that vector to access to your local network in other ways. They don't need Tailscale's SSH web client for that.
A bad Chrome extension does not allow the bad guys to open a terminal on my machine, load my ssh keys, launch an authenticated SSH connection, and launch an authenticated SSH connection into an enumerated list of remote servers.
literally everyone?
I'm aware that web-based email and credential managers exist, but GP asked "...why would it not be able to access your email or password manager?" I answered that, with my app choice, I don't see how they could.
It is not a web page with a shell open.
It is the Tailscale client, compiled to WASM, maintaining the keys for connections to nodes on the tailnet. Connections opened from the browser engine don't get the ability to reach the tailnet.
> Connections opened from the browser engine don't get the ability to reach the tailnet.
That was not really explained in the article. Maybe it's obvious to some people, but I'm sure it's not for many.
Or just use userify's public key distribution model (then you can stick with a simple and proven design that just automates the standard SSH design that's been around for decades)
Or just do it the old way, by hand: drop your teams' public keys on groups of remote servers, setup their sudo access, and you're done.
Or just use Ansible or Chef or Terraform, or any of those other server orchestration tools, too (just without the permission layer/bulk remote session kills/user deletes of Userify).
of course, if you really need an ssh terminal in a browser, then you have to go with one of these sorts of crazy things, and at least it's not using passwords!
Let me say again, though I admire it, I'd never use this. I like to sleep soundly, as irrational as that may be.
I think this should be compiled with WASM and deployed via web page. Then we can explore the idea of browser-hosted POSIX kernels.
I love that they were clearly inspired by fly.io. Warms my heart that a random blog post with a good idea can spread like this.
But yes, we love Fly and use them (and they use us) and we share a slack channel between our two companies for casual banter.
https://lists.zx2c4.com/pipermail/wireguard/2021-January/006...
Forgive my ignorance but is there any sort of native client besides the browser running in the background to help with websocket to tcp? Or a tunnel to a cloud service to help there?
However on the bright side performance.now() has a much worse resolution on most browsers, making timing attacks harder.
Definitely not for browsers as the execution environment, at least browsers running extensions.
Nowadays, it's a lot harder to get past browser security and people run all sorts of applications in browsers such as banking, business critical SAAS, email, etc. So, perfectly fine to include some crypto in that and probably not optional to do so for a lot of applications.
What's still true is that you should not be rolling your own crypto libraries and instead use libraries from reputable sources that have been scrutinized by people that know what they are doing. That's true whether you use javascript, C or whatever. And of course with web assembly you can just compile those libraries and use them in a browser sandbox.
One of the things that Teleport lacks (IMO) is Wireguard
Having an SSH client in your browser join your VPN violates all the principles of modern computing.
I think that's a compliment? You're welcome? :)
edit: _especially_ when it comes to security
I used to think that everything in the world should move to the browser (in my case, that would be high performance molecular graphics and microscope control) and ChromeOS was the logical extension. Web Assembly to handle the existing C++ codebases, a collection of standard web tech to handle the user interface. Big fan of SSH extension because it meant that machines that only ran a browser could be useful command line programming terminals. But didn't like that SSH extension was written in a dead-end container technology (NaCl) and the CHrome folks sort of messed up multiple ways for the extension to integrate better.
But after working with web tech for enough time I came to conclude that putting things in the browser like this is an antipattern, in particular increasing the surface complexity of the software space while not actually replacing existing systems (openssh continues to exist, OS-level VPNs continue to exist, even after you port the VPN and client to web assembly and run it in a browser).
In my mental model, it makes more sense to put the browser in a VM and then wire the VM's networking to a VPN handled by the OS, rather than putting what is more or less a significant fraction of a virtual machine manager's capabilities into the browser. If you're going to do that, why not go whole-hog and add a VM to chrome so that it can run linux with a full networking stack and then host an SSH client inside that? So you can run linux in your chrome in your linux.
My compliment is: I am impressed at how well you parlayed several technical projects into a thought leadership position, but we have fundamentally different architectural principles and work in different domains. Your work disrupts mine, but mine doesn't disrupt yours. My enterprise actually disallows me from visiting your company's website on my work computer because users installing their own VPNs is considered a security risk (fwiw, I bought into BeyondCorp, which eschews VPNs, a long time ago, and would prefer my enterprise eliminate VPNs, as they don't really protect our users).
- An SSO-authenticated web interface, integrated with a host agent on your instances, means you don't have to manage SSH keys.
- If you just need a disposable CLI that inherits permissions from your SSO-authenticated user role, you can do that from a disposable box in a web interface after authenticating via the web interface. Google Cloud Shell is a good example.
- Cloud-native development is easier if developers can just start working on a unified environment, without having to set up & maintain a local environment. Utilizing the web browser avoids the need to consider separate tools and separate methods of network connection.
- SSH'ing to "private" instances is impossible without going through a bastion or VPN. The bastion then becomes a single point of attack, and is hard to maintain and secure. Similarly the VPN is an additional attack vector, maintenance headache, and requires client-side software, configuration, troubleshooting. Instead of deploying a bunch of bastions or setting up a VPN, if you can use the backend control plane through an SSO-authenticated API gateway, along with a backend proxy to internal networks, you can avoid bastions altogether. This is the best practice for Zero-Trust. Google Cloud IAP Proxy is a good example.
The implementation of it, with WebAssembly/WebSockets/WireGuard/DERP, may be lamentable for several reasons. But it probably (I assume?) solves problems that other SSH Web Interfaces didn't. I hate that the web browser has monopolized computing interfaces :) But in this case it seems to solve many problems.
Turning this around. Let's take the idea of using WASM to put a full environment in the user's browser. This is a logical idea, after all- WASM exists to make it possible to write applications in Not-Javascript and deploy them in a browser. IE, don't stop with SSH: you should have a web server, a shell, multiprocessing, scripting languages, everything necessary to host VSCode server and a self-hosted compilation toolchain in a browser. Full linux user space in a browser, enough to compile ChromeOS and boot into a browser running linux
What have you achieved? A very expensive (in terms of porting cost, CPU usage, and deployment size) inner platform that does what an OS does already. But it's inside the browser, with a patched version of code (because WASM always trails native apps), with each sub-application maybe linking in its own TCP stack. So it will always trail innovations in desktops, since it's not a full replacement for the existing system. So it makes the world more complicated and exposes more surface areas for security management.
"""The inner-platform effect is the tendency of software architects to create a system so customizable as to become a replica, and often a poor replica, of the software development platform they are using. This is generally inefficient and such systems are often considered to be examples of an anti-pattern."""
Like I said elsewhere, I'm not completely opposed to the idea of the browser as a complete and fully functional application container for an inner platform. And I want to encourage creative people to try new technologies, especially WASM to explore the idea of "how much can we move to the browser". However, I see the container as the mediator of the network, not the application.
Give me a stream of bytes and let the whole world see it for all I care--wireguard will build a secure private network entirely on that stream of bytes. It could be totally public coffee shop wifi with zero encryption (basically yelling your passwords and secrets out in the open) and yet wireguard will make it secure and private for me.
So in this case who cares if its a websocket to the browser vs a 'proper' (bloated, shitty) OS VPN. Give me a stream of bytes and I'll build my own secure and trusted network on it thank you very much.
I understand the desire for moving more TCP logic to applications but, given my experience with network technology, I would predict that ten years from now, people will hate the experience of having to update 30 apps to get 1 fix to TCP performance that would have just been a kernel upgrade. IE, like everything that happened with the web and inner platforms, it's more work, doesn't replace the existing system, and just makes the admin's life harder for the ostensible purpose of being more convenient for the developer on their own machine.
It gives a new meaning to "my computer is a datacenter" phrase.
we easily run multiple VMs kn the same laptop with multiple kernels with multiple TCP/IP implementations; even withing the same OS user space programs come with their own DNS resolvers that ignore system wide settings (Go without cgo, I'm thinking of you), and soon we're going to have a proliferation of user space networking stacks.
On the other hand, if you believe in the future of OS private network connectivity -> console -> ssh, then you had that already with native Tailscale and Tailscale ssh.
If you believe in OS private network connectivity -> browser -> javascript console -> ssh, then you can do that too, by installing tailscale in the native OS and then the browser can use it.
I actually agree with you, I'm very suspicious about a world where we just move everything into the web browser. But on the other hand, sometimes it's really handy to have that option.
To be honest I can't evaluate your product at work- to determine whether it helps our users and whether the idea of moving more of the network stack into the application makes sense- because my corporation (a large multinational pharma) disallows us from visiting the entire tailscale website because you sell a VPN product(!) which isn't our standard one. I'd love to change that policy but I'd still want to move to a BeyondCorp world (https w/ auth), not put a VPN in my browser. Or make Tailscale our standard VPN.
I see the point of "it's really handy". That's how we got Javascript which is a cost we now all have to pay.
Tailscale is not a typical VPN; it's just a system that attempts to provide beyondcorp-like behaviour at a lower level of the stack, so that you don't have to rewrite all your apps (ssh in this case!) to use https, and don't have to have open ports in your firewall, and don't have to run everything through the cloud if you don't want.
As in my post above, there's more than one way to do it. You can also build traditional-beyondcorp-over-https on top of a Tailscale network, so you get all the improved network connectivity and also all the benefits of a "pure" beyondcorp architecture.
> Web-based SSH clients aren’t new. Nearly every VPS and cloud provider already lets you connect to your VMs from the web — so how is this different?
This is clearly isn't for everyone, but if you need or already use something like that, then I think this has a chance to be more secure than some other options.
The issue is wiring up the browser-hosted application with a custom network inside the browser. It's a truly interesting but highly disruptive concept and I'm curious how it will play out. Perhaps in the future every program will statically compile its own TCP stack and talk over RAW sockets, but... that's sort of throwing away everything BSD and Linux and Windows achieved over the last few decades in terms of OS abstractions.
> Perhaps in the future every program will statically compile its own TCP stack and talk over RAW sockets
A surprising number of common applications already do this.
I ran into a recent bug[1] caused by a Windows update where TLS handshakes would randomly fail. But this only presented in a few apps. Browsers and .NET apps were all completely unaffected because they don't use the OS level functionality to handle TLS.
[1]: This is a link to the KB that fixed the issue, which was introduced in the 22H2 cumulative update (search for "SEC_E_ILLEGAL_MESSAGE"): https://support.microsoft.com/en-us/topic/october-25-2022-kb...
TLS is different from TCP. TLS support might be provided by an OS, but it's certainly something an application can link in since it's really just a byte translator with some additional complex logic. TCP is an OS-level protocol for all the reasons that history chose it (having your network device and network protocol in the same ring).