Tailscale SSH | Hacker News Reader

Tailscale SSH

tailscale.com

759 pointsignoramous303 comments

I'm one of the authors of this. Happy to answer any questions.

One of the fun technical details is that, when enabled on a machine (tailscale up --ssh), the userspace tailscaled process takes over all TCP port 22 packets after the WireGuard decryption and doesn't even feed them into the kernel over TUN. We use gVisor's netstack to handle the TCP connections in-process.

So it doesn't matter whether you have other processes (or iptables rules, etc) that would prevent the Tailscale SSH server from binding to port 22. This lets people gradually use Tailscale SSH over time without messing with their system one.

The Tailscale SSH server currently only runs on Linux but there's support in git main for macOS too but it's not super well tested yet and not included in the sandboxed GUI builds currently.

> This lets people gradually use Tailscale SSH over time without messing with their system one.

That is something I have really appreciated about Tailscale. It seems to consistently not mess with the existing environment. Considering it does networking witchcraft and it works on a variety of architectures and OSs this is quite an accomplishment.

I suspect Tailscale's customers have found the same.

That is not a feature it is a bug and a big hole.

The firewall is the system. Just like apple bypass its own firewall and just send packet back home. Or the chinese way.

Of course as said by one of the author the key is to control port 22 or rule for ssh. That is not a totally lost. Still, one that is ok … you are breaking the system by promoting a way to bypass it. Or just 1 rule. It is so hard to remember.

No, it's not. Network access control is the whole point of Tailscale; it is the network filtering layer. It serves literally the same function that a Checkpoint Firewall-1 installation would have in 1997, and that's why people buy it. This is basic stuff from the Tailscale website; it doesn't even qualify as analysis. You really ought to understand how these things work before you describe things as "big holes".

Because that's what we all want. Yet another place to look for ACL rules...

If you're deploying Tailscale? Yeah, that's about right.

Considering how simple it is to use Tailscale ACL rules with node auto-tagging, yes I absolutely want it.

Anyway there's a loophole on your network. Tailscale is just a way to use it.

Not really. It messes with DNS big time. Try enabling the "MagicDNS" or "Exit Nodes" features, and watch as /etc/resolv.conf is edited with each change. I can easily reproduce scenarios where it's left empty and there's no working DNS resolution.

This is one of the major things I _don't_ like about Tailscale. I wish they'd just stick to enabling Wireguard and making the authentication easier (i.e., where they started). I'm not a fan of most of the features they've added since. I don't want service discovery, magic DNS, SSH key management and/or the kitchen sink bolted on.

Consistently I’m unable to use Tailscale on a GCP instance and also use GCP services cleanly, because it messes with the DNS route to the metadata server. Otherwise, it’s a great product.

Thanks for the feedback. I've filed https://github.com/tailscale/tailscale/issues/4911 to fix that.

https://github.com/tailscale/tailscale/issues/4911 is now fixed and will be in the next release.

I don't use GCP, but this is a high quality example of a company doing feedback right. Nicely done!

It only messes with /etc/resolv.conf if you did `--accept-dns` and don't have systemd-resolved, which nowadays is much more common.

Linux DNS is a clusterfun: https://tailscale.com/blog/sisyphean-dns-client-linux/

But, yeah, without systemd-resolved Linux DNS is a fight for the death between uncooperating processes. NetworkManager is okay but there are a dozen buggy variants in the wild we have to work around.

Linux is by far the worst platform for DNS config.

I totally recommend systemd-resolved. It's the only thing that does DNS well on Linux.

What about using NSS[1]? You could add a Tailscale provider to the `hosts` entry.

[1]: https://en.wikipedia.org/wiki/Name_Service_Switch

This is neat. I've used Cloudflare's Zero-Trust SSH, but I've been frustrated that it interacts poorly with sftp and scp because of the client-side changes that they make to ~/.ssh/config

Does tailscale have the same issue?

We don't modify or require changes to your SSH client. You can use any SSH client you want.

Tailscale employee here. Tailscale SSH works at the target side by listening on the SSH port on that machine. Client changes aren't needed for this to work, all that is required is to use your SSH client as normal. This should allow you to use sftp and scp without issues.

For what it's worth I encountered the same issue and came up with a solution:

https://github.com/cloudflare/cloudflared/issues/574

Cloudflare have ignored the github issue (which includes a solution) but at least 3 other people seem to have found my solution helpful.

Thanks! Make that 4.

This looks great, and I'd love to replace AWS SSM (at least for the purposes of instance access) with this! One question I have is have is around device limits.

With SSM, I can easily run an agent on every instance. Tailscale has pretty tight device limits on the Team and Business plans. I have no idea what the custom pricing looks like, but I'm guessing it would exceed my budget. What's the intended way to use this with a large number of servers? A small team can easily have more devices than 5x or 10x the number of users. Should we just set up some "gateway"/"bastion" instances to access via Tailscale SSH and then use regular ssh from there? Some sort of more limited device mode that doesn't count against the device limit (for ssh only, perhaps?) would be great.

You could do a Tailscale SSH bastion thing, yeah. But before you build a funky setup to avoid pricing concerns, at least reach out to the sales folk to see what it is. We're usually pretty flexible on exact quotas and realize that different orgs have different user/device shapes.

You'll want to set up a bastion host as a subnet router. https://tailscale.com/kb/1021/install-aws/

> (...) the userspace tailscaled process takes over all TCP port 22 packets after the WireGuard decryption and doesn't even feed them into the kernel over TUN. We use gVisor's netstack to handle the TCP connections in-process.

> So it doesn't matter whether you have other processes (or iptables rules, etc) that would prevent the Tailscale SSH server from binding to port 22.

This sounds like a great feature when exploiting buggy WordPress/php apps! /s

I realize this is a feature - but it's a bit sad that the standard package handling isn't up to the task; leaving (I expect) the tailscale daemon as a "magic" netcitizen - not featuring in neither "ss" or "iptables" output (why can't I login to opensshd?).

How do you figure? The idea is that Tailscale is bypassing the kernel, which it can only do for requests coming in over the tailnet --- it gets those packets raw, directly from WireGuard, unlike the normal IP packets your kernel routes to/from localhost or an egress interface.

It's not the same, but "Docker Network bypasses Firewall, no option to disable" https://github.com/moby/moby/issues/22054 (2016)

You're right, it's not at all the same. The Tailscale bypass exists (1) only for traffic traversing Tailscale interfaces (by design, that's the only traffic it can impact, because Tailscale can't run a userland TCP/IP stack for non-Tailscale traffic), and (2) only for this one feature, and (3) only if you've explicitly allowed it for particular users in your Tailscale ACLs. It's not clear to me how you could screw it up to, e.g., amplify an SSRF attack.

All I'm trying to point out is that advertising "this bypasses the firewall, by design" has been abused in the past.

[edit] It boils down to principle of least surprise, managing expectations, etc. - proper documentation is indeed key.

This sounds like more of a marketing thing, where the wording needs to employ care?

You have to explicitly enable Tailscale SSH, both on the host and in the ACLs that allow users to use the feature. Tailscale's ACLs are much, much better than iptable rules (for instance: they have built-in unit testing).

(I'm not impartial about Tailscale.)

> Tailscale's ACLs are much, much better than iptable rules (for instance: they > have built-in unit testing).

Humility helps a lot on the internet- the important thing about iptables is that it runs on millions, possibly billions of machines. Production systems that don't have unit tests but run at scale aren't worse than systems which are newly introduced but have fairly unknown implications.

I'm sorry, I really don't know what you're trying to say here. I'm evaluating a set of engineering tradeoffs and reaching a conclusion about them; I'm not trying to psychoanalyze them.

Im trying to say you're better than iptables because your code has unit tests makes you look arrogant because iptables is a production system that operates successfully at such a large scale that it shows unit tests aren't an accurate measure of quality. I'm saying that when people talk like you did and criticize prod systems, you look arrogant, and humility- using terms like "we believe" rather than "is" help a lot in building user confidence.

Again: these are engineering details, not people; they aren't "arrogant". There is lesser engineering and there is better engineering. As someone who does quite a bit of work with iptables and who has used ACL systems like Tailscale's, I can tell you right off the bat that Tailscale's system is better, and if you have the option of using one or the other --- there are good reasons you might not be able to --- you should use something like Tailscale's, which is identity-aware, testable, dynamic, and simple. Obviously, if you're not using Tailscale at all, this is a moot point, for many reasons, including the fact that if you're not using Tailscale, you don't have to think about how it interacts with your iptables rules.

I'm not making a value judgement about people who need to keep using iptables. I might be making a value judgement about people who demand that everyone else keep using iptables.

OK, you're free to completely ignore my advice that you look arrogant, and that it might affect the uptake of your product from the very people who could lead the way to increased adoption.

But, my point still stands. You can't simply assert your system is better, it has to be proven at a scale similar to iptables before you can say that.

It's not my product.

(I'm not impartial about Tailscale, but I don't work there).

> [only for..] requests coming in over the tailnet

Well, that's certainly different from "all TCP port 22 packets" - I suppose some emphasis should be on "after the WireGuard decryption" (ie: over the wireguard interface). It's not entirely clear from the comment (but probably clear to engineers working on the tailscale code).

I read it as if tailscale snapped up packets before the kernel from (all) network interfaces...

How could it possibly work otherwise? Tailscale owns the WireGuard connection, so it gets raw packets from WireGuard before the kernel.

It could work like a full userspace network stack, getting the packets on the wire before (instead of) the kernel (network stack)?

How? (I can think of ways, and they're all horrible).

The fact that all the ways are horrible is indeed why I was so surprised to (mis)read tailscale as having such capabilities... I'm happy I misunderstood.

Hi Brad: Thanks for helping out with this feature! I've been one of the early users of tailscale. My network is around 50 machines. I've recently started having issues with ssh on some of my machines, especially from mac m1 -> some ubuntu boxes. Could this be related to this new feature? Any suggestions/pointers on how to debug these issues?

Tailscale SSH doesn't mess with your port 22 packets if it's off so almost certainly unrelated. Have you reached out to support or filed a bug?

Just a quick thank you to the team working on Tailscale. It’s hands down the most seamless dev experience I’ve ever seen. Every time I think “such and such would be nice”, I search the docs, and it’s already implemented better than I could have expected (eg the stateless mode for ephemeral servers).

I tinkered with cloudflare before that but just couldn’t get on with the interface of the admin tooling.

With Tailscale I have a lot more confidence that I’ve set up the access rules as I need them. It’s all just a lot more obvious.

Could you share some details about the embedded SSH server? I'm curious if this would work to add SSH capabilities to devices that run Tailscale but don't include a built-in SSH server. Previously I've used dropbear, so it'd be really nice to be able to drop that requirement!

If you're already running recent-ish Tailscale on them, they're already running an SSH server that's just disabled. Run "tailscale up --ssh" to turn it on.

The code's at https://github.com/tailscale/tailscale/tree/main/ssh/tailssh for all the details. Which details in particular are you curious about?

Ah it's using crypto/ssh to provide the SSH server, so I think that's all the details I needed! Can't wait to give this a shot.

> Which details in particular are you curious about?

In the linked Q&A video with Maisem, you spoke about buying books on Linux to make SSH work. Which books, if you don't mind me asking?

The Linux Programming Interface: https://man7.org/tlpi/

Advanced Programming in the UNIX Environment, 3rd Edition: https://www.amazon.com/gp/product/0321637739

Can you do ssh tunnelling?

You can do local port forwarding today, remote port forward is still a WIP. What do you want to use it for?

Disclaimer: I am one of the engineers who built Tailscale SSH.

Accessing Clojure nREPL servers, local port forwarding works for that. This allows me to inspect and modify running applications, a Lisp superpower.

hello, is sftp supported? thanks

Yes: https://github.com/tailscale/tailscale/blob/v1.26.1/ssh/tail...

What would we need to have open in our security groups for this to work?

I think ingress wouldn't be necessary since tailscaled creates a tunnel right?

But how about egress traffic? UDP for WireGuard or something else?

Security groups where? On the Tailscale ACL side, you need to allow tcp/22 in.

On your host where you're running Tailscale, usually nothing. You can keep everything locked down for ingress. Outbound UDP only, but usually cloud VMs allow outbound traffic already. (This is covered more in https://tailscale.com/kb/1082/firewall-ports/)

I meant AWS, that page explains it all, thanks!

quick question. Does this do user (de)provisioning like Jumpcloud? I.e. if the target machine doesn't have a /home/someuser but someuser is in my tailnet ACL, will it create the account?

From [1]: “Like other SSH clients, Tailscale will only use user accounts that already exist on the host, not create new accounts.”

[1]: https://tailscale.com/kb/1193/tailscale-ssh/#ensure-tailscal...

A nice next step would be tailscale managing an ssh key that's allowed to do interact with a git(hub) repository. So that I wouldn't have to create multiple keys or setup the same key on different machines and still be able to interact with a repo from all of them.

It'd be really nice just using git transparently and having tailscale take over the git ssh connection and authenticate using taliscale access controls.

At least for personal projects or small teams that'd be quite convenient.

Depending on which part of those things you find painful, you might want to look into ssh certificates? They're pretty easy to work with, much easier than most kinds of certificate systems.

Just want to say thanks: This is insanely cool/easy. Combined with the VSCode Remote SSH extension and MagicDNS, it's now insanely easy to work on a project in a remote environment. I was recently reading through a relatively long post on setting up SSH through Tailscale to access a WSL2 environment, and now it's literally as easy as popping open VSCode in any environment that I have Tailscale installed on and accessing `user@my-magic-dns-machine`. Great work!

So we have no way to secure this besides disabling wireguard ?

There's no "disabling wireguard" in Tailscale unless you don't run it at all.

You can secure this by:

- Not enabling the SSH feature on hosts where it's not needed - Creating ACLs so only certain clients are allowed access.

So essentially, just use the same mechanisms as for everything else in Tailscale.

How was the decision made to roll this functionality out before announcing it to customers (we found it during a previous security audit)?

While it might seem logical in your mind to bolt on extra features and add value, your customers evaluate risk based on functionality of the software they are approving. Customer buys a VPN solution, magically gets remote access that bypasses firewalls. Can we trust Tailscale to not roll out a remote file backup feature and start silently exfiltrating data (as an extreme example)?

There are two things have have to be enabled to turn it on:

(1) a target server needs to run "tailscale up --ssh" to enable the SSH server

(2) your Tailscale ACLs have to permit it. Our default, if you've never set your ACLs (as is usually the case for personal users), is that you're allowed to SSH to your own untagged devices only.

For an org that's already using ACLs, you won't have any SSH rules defined and thus nobody in your org can enable the SSH server. (Or rather, they can enable it but nobody can connect to it.)

If your concern an org that's using the default "all packets are allowed" ACLs?

Concern more around what looks like an ssh backdoor showing up unannounced. How would they know the subtleties of what it takes to enable it when it wasn’t announced yet?

That's one part of it.

I can't speak for mike_d specifically, but there is a concern with having (potentially significant) modifications made to the codebase that aren't surfaced in the release notes. I imagine closed-source projects do this on a regular basis whether customers know (or care) or not.

The expectations for opensource projects are different though, particularly when it comes to system-level or near system-level components. So not being able to access the functionality is a great default but it doesn't address side effects of the changes or the desire to know about changes being made in our environments.

I’d think that the literal ability to audit the source code would satiate one's desire to know about the changes being made in their environment.

Of course it doesn't. Only the actual auditing of the code could do that. Nobody in the world wants to audit code they are relying on every update to make sure that the developers have not added potential new security concerns that they would otherwise not be aware of.

Try to look at it without your inside knowledge of how it works. Think about a customer discovering this with no documentation.

Until you decide to ship a completely on-prem Tailscale server, ACLs mean nothing. They can be modified by the same rogue employee that added an SSH server that bypasses local firewalls to our environment without telling anyone.

If you're unwilling to trust Tailscale and their processes, you can't run Tailscale right now. That's obvious. It's part of the premise. The idea that ACLs "mean nothing" is risibly reductive; the ACLs protect our team members from each other and mistakes they might make with their environments.

(We don't use Tailscale SSH, and are unlikely ever to; we have a separate source of authentication truth for SSH, and a separate certificate-based access control system.)

They built a footgun into a toaster, and victim blame when people complain that they thought it was just supposed to make toast. Users should not be put into a situation where they need to configure ACLs in anticipation of undocumented features.

My hope was that with a little public prodding they would do better in the future. It is a product I want to like, maybe not for what you or I do, but lots of folks out there are slinging cat pictures where it will be a net benefit.

"Victim blame". They announced an opt-in feature.

Ah, I see the part you are missing. It was rolled out before it was announced or documented. We found it during a pentest.

Ohhh, this explains why my corporation placed a total firewall block on the Tailscale website.

This is a postmortem-worthy incident on Tailscale's part.

It's likely just because it's a VPN not under control of the corporation. Corporations have this magical wand which they swing to make it hard for people to do their work :)

If you're not already using Tailscale, with your security or IT teams controlling it, it would be malpractice to allow it on a controlled network. No competent security team allows people to introduce their own VPNs.

Speaking of which, how would they detect it?

Any good network security monitoring system should allow it to be fingerprinted in some manner, and if deep packet inspection is in use then it should be blocked outright.

Blocking the tailscale website has nothing to do with blocking the VPN. I know you're passionate about them, but I think you got baited here...

Who’s talking about blocking a website?

dekhn, three comment levels above yours.

You're right. I guess my brain wouldn't let me process something as dumb as a corporate security control based on blocking a website to keep people from installing a binary.

Anyways, I'm just here to say, corporate security teams are definitely not OK with you doing a rogue Tailscale install, and that's as it should be.

I work for a Fortune 100 company and this is precisely what our corporate overlords do with non-approved VPN software. There are device management tools on every machine in the corporation to detect and block the software, but that isn't turned on. Just blocking the website. ¯\_(ツ)_/¯

Anyways, I'm just here to say, corporate security teams are definitely not OK with you doing a rogue Tailscale install, and that's as it should be.

You might be shocked at how often I get "can you deploy a tarsnap server on port 443? My company's security team won't let me connect to your server on port 9279" requested.

I mean, it's trivial to bounce the TCP connection... but I'm not going to help subvert security policies.

The way it works in enterprise that is principal engineers like me are generally given some freedom to explore new technologies responsibly. In my mind, that includes visiting the Tailscale website (which started being blocked by our IT yesterday) to gather information about whether this would be a good alternative technology for our research teams.

Now what I have to do is file a bunch of tickets and take a bunch of meetings to get a block removed from the overall site. Really, what I was trying to do is provide nformation to the Tailscale developers that enterprise already considers their website/product scary enough to do a whole block, and if they want to expand into enterprise, they may want to understand the reasons for that.

> Now what I have to do is file a bunch of tickets and take a bunch of meetings to get a block removed from the overall site. Really, what I was trying to do is provide nformation to the Tailscale developers that enterprise already considers their website/product scary enough to do a whole block, and if they want to expand into enterprise, they may want to understand the reasons for that.

Not all large enterprises are this disfunctional. I'm sure Tailscale are doing just fine.

I have Tailscale on all my Macs. I use MacOS default SSH between my machines, but only via the Tailscale interface.

Nevertheless, I had to open SSH on each machine, and it's a nightmare to close up the firewall so only Tailscale gets through. You'd think this was the whole point of Tailscale; there should be a one click lock to restrict to Tailscale. But the Tailscale documentation is wanting. I actually paid for a candidate for the best firewall front end, it came with "Let us know if we can help!" and radio silence once I explained the problem. Likely, restricting to Tailscale requires a granularity one can only hand-code.

I can write a firewall, I've written plenty in the past, I just couldn't find the several hours to do this as a one-off for me when it should be easy, but I was missing needed information.

Tailscale is justly proud of how it connects machines through uncooperative routers and such. Tailscale SSH should do the same. The idiot's guide to securing a machine so only Tailscale SSH gets through should be to find SSH in the preferences and turn the fucker off.

You don't really need a firewall to do that. putting

  ListenAddress 100.x.x.x

where 100.x.x.x is the address on the tailnet, into your sshd_config would do what you want. Unfortunately you can't specify an interface, but if you have any sort of automation in place this is easy enough to template in.

> Tailscale SSH should do the same.

It does. You can "turn the fucker off" (as you say) at the OS level and Tailscale SSH will still work. We don't send the Tailscale SSH packets through the OS for it to block them.

Well, Tailscale SSH server support for macOS is still not entirely done. You can build it from source if you're brave (and set an env var to turn it on), but it's not in the product yet by default.

Any interest in adding mosh like features (https://mosh.org/)?

Low latency typing, session resumption etc

Tailscalar here, it should just work using the normal ssh bootstrapped method.

I've been using Tailscale for years but will likely not use this feature, even though I would like to.

The fundamental problem with the approach really is that connections are different over the tailnet and over the local network. Here is a specific use case that is painful:

1. There exists a cluster of machines, each with large amounts of locally attached storage. They are all on the same local network and connected with 10Gb (and likely soon 40Gb ethernet interfaces).

2. Each machine is individually on the same tailnet so they can be accessed remotely.

3. Remote users frequently need to move large amounts of data between machines. A user copying a few hundred gigabytes of data with "scp" is normal.

4. For performance reasons, it's preferred to avoid the Tailscale/wireguard overhead when copying data between adjacent machines in a rack.

At this point, if I enable tailscale ssh for remote login, it appears that the problem of key management for connections between local machines (using ssh over the normal interface, not the tailnet) still remains, and in fact, the overall authentication configuration is more complex than it was before.

What I would love to exist, and would make me instantly use this feature, is if the tailnet issued SSH certificates (probably injected into its own ssh-agent?), the existing tailscale SSH implemention worked just like it currently does (it's great!), AND I could manually configure servers to accept certificates issued by the tailnet. Then SSH paths like "laptop --> (over tailnet) --> server 1 --> (over local network) --> server 2" could be made to work transparently, for those machines that need it, and for regular users, it still "just works".

I agree that'd be fun. We have something similar in the works for other protocols, but maybe SSH isn't a huge stretch to extend it to!

Does the current setup with magicsock mean that tailssh behaves similar to MoSH (in dealing with resuming a session, specifically)?

Yes.

But so does regular SSH over Tailscale, so Tailscale SSH isn't special in that regard.

Oh that sounds exciting, would it also solve the current performance issues when moving large amounts of data? It's currently the only reason I still have to use public IPs for some applications.

If those machines are in the same rack, why you don't put them on the same subnet and use a different interface when moving files around instead of Tailnet?

That's exactly what we do, which is why adding Tailscale SSH to our current workflow isn't helpful, since we would still have to manage SSH keys for access via the local subnet.

Thank you to the Tailscale team for altering my belief that VPN could only stand for Vexing Productivity Neutralizer.

Features like Tailscale SSH represent the ruthless removal of annoyances.

edit: grammar

Hey bradfitz, guy who previously had 32150 here. :-) This looks insanely cool, a couple questions:

I know it says it's linux-only right now, but is that client side or server only? Can my Windows users TailSSH into linux boxes?

Would be cool if somehow it could wedge into sudo auth so you could login as a a user and sudo without password if allowed by ACLs, especally if I could add "check" to the ssh. agent pam module?

One thing that has prevented me from trying Tailscale, despite the great word on the street, is I can't figure out pricing, despite contacting sales. I'd like to run it on ~120 dev+stg+prod VMs, with 10 people (devs, testers, ops). I'd like every box to talk over tailscale directly, as an overlay network, but servers I hope aren't users, that'd get expensive fast. But I need more devices than 10/user. I presume "custom" would help with that but I got no reply from sales. We are probably too small fry. Now that I'm typing this, I realize I guess we could just buy ~15-20 users despite needing only 10.

I think I've resolved myself to setting up Nebula for the server overlay network, and using Tailscale for physical users, with a traditional firewall bridging them.

Again, Tailscale SSH looks very nice, job well done!

> I know it says it's linux-only right now, but is that client side or server only? Can my Windows users TailSSH into linux boxes?

Linux-only on the server right. macOS support is kinda there (in git) but not entirely done and not included in the GUI builds. Windows server support is tracked in https://github.com/tailscale/tailscale/issues/4697.

You can use any SSH client from any OS.

> Would be cool if somehow it could wedge into sudo auth so you could login as a a user and sudo without password if allowed by ACLs

Some of the start of that is in https://github.com/tailscale/pam

> One thing that has prevented me from trying Tailscale, despite the great word on the street, is I can't figure out pricing, despite contacting sales. I'd like to run it on ~120 dev+stg+prod VMs, with 10 people (devs, testers, ops). I'd like every box to talk over tailscale directly, as an overlay network, but servers I hope aren't users, that'd get expensive fast. But I need more devices than 10/user. I presume "custom" would help with that but I got no reply from sales. We are probably too small fry. Now that I'm typing this, I realize I guess we could just buy ~15-20 users despite needing only 10.

You only pay for unique humans, not tagged role account devices. I wonder if your email got eaten as spam or something. Email me (username at tailscale) and copy sales@ and I'll make sure somebody replies. But I don't think you need a custom plan.

> I think I've resolved myself to setting up Nebula for the server overlay network, and using Tailscale for physical users, with a traditional firewall bridging them.

Hey, if you've got something that works, stick with it. :)

> You can use any SSH client from any OS.

I've tried this earlier and was unsusccessful sshing from my iPad, using Termius and Blink apps. Not sure if there are specific client requirements on the iPad?

We successfully tested a number of iOS SSH clients. They should all work.

Can you file a bug with details of what you saw? Either https://github.com/tailscale/tailscale/issues/new or email support@ ... whichever you're more comfortable with.

sure thing, will do!

ah I think I figured it out!

I had a notification asking me to verify, but because of Focus, that notification didn't show up anywhere that I could see...... So in theory, this should work, will try again.

Works fine now

If you only pay for unique humans, why does the pricing page list device count caps (and, on Business plans, the pricing for exceeding those caps)?

Because if you are an IoT service with one human and 100,000 devices, the amount of support you may need is more dependent on the 100,000 than on the 1. Very large numbers of devices per human need somewhat different pricing.

Very promising the start of the pam. Good news about the SSH client, I figured that was the case but wanted to ake sure. That would be a huge benefit for my developers who are all on Windows.

Thanks for the info about pricing, I set up the 1 user free account and started that to get some hands on experience, and I'll copy you on pricing if I can't get it figured out. Thanks!

Just to add to the above, pricing was a little obsecure for me too though I commited to Tailscale and then worked it out after the fact.

Minor suggestion, for future and new users, is it possible to get a calculator where you could input the number of users you expect, the number of servers you want to include, expected unique ACL's and provide you an ETA of what your license cost would be?

I've passed that on to coworkers.

Regarding pricing, in my experience the Tailscale crew have been very forgiving when it comes to user/device limits. I'm sure if you have 10 users and 100,000 devices you would get some attention, but keep it reasonable and you should be OK.

What are the failure modes? Openssh is a well understood risk, this seems... unquantifiable?

Tailscale is a userland process built in a memory-safe language, which leaves you only the SSH protocol cryptography-type vulnerabilities, which are themselves mooted by Tailscale (see downthread for a discussion about why they didn't simply expose rsh instead of ssh).

This is safer than OpenSSH.

(OpenSSH, as a piece of software, is extraordinarily safe, and has one of the best records of any memory-unsafe codebases. But OpenSSH as configurable infrastructure is much less safe; people screw it up all the time.)

Sure, but how reliable is it. What are the risks that I'm going to get locked out of a production instance and suffer loss of earnings?

Do you build tailscale with redo? and if not why not? :)

Were golang’s ssh and crypto packages independently audited ?

I attempted this on a VM inside a Linux host and got a lower privileged user from inside the guest VM to ssh to a root-privileged user outside on the host. Both were authenticated to Tailscale with the same gmail account, so from an OAuth perspective, this is valid. From the OS perspective though, the host SSH port is blocked, and a guest should never get full root access to the host or see the host's resources.

I am not sure if I am confused about something, or maybe there are prod use-cases where the same IDP identity should have different roles/privileges depending on the machine, and Tailscale SSH breaks that?

I've been having trouble adopting Tailscale. As so many others say, relying on another identity provider is unfortunate - I, too, worry what happens when Google decides to lock me out because some algorithm decided my account is fishy.

The biggest blocker has been the issues with the Android client. I'm either hitting https://github.com/tailscale/tailscale/issues/915 or https://github.com/tailscale/tailscale/issues/4611, but neither issue appears to have a fix coming soon. Whenever I am on my carrier's network, my phone's internet stops until I disable Tailscale - that's just a show stopper from using TailScale.

So instead of developing this SSH feature, I would have preferred to seen them work on their bug backlog.

In the meantime, I'm experimenting with ZeroTier. While it doesn't have the ease and cool magicDNS+LetsEncrypt feature, I think I'll survive with something more reliable.

Are you after the LE part specifically? If not, I'm quite happy with mdns and the seems to be a unicast version available too:

https://www.zerotier.com/2021/05/06/zeronsd-unicast-dns-reso...

For public domains, I've got a quick script which mirrors what appears in avahi to route53, so that's one way to deal with certs.

I appreciate that Tailscale runs the DNS server so it's one less thing for me to manage. Similarly, the built-in LE is just icing on the cake as it's one less thing to think about. Once https://github.com/hassio-addons/addon-tailscale/pull/89 is merged, running Home Assistant on a VPN with a LE certificate, would be such a quick setup for anyone.

Indeed, you can do all that yourself as you point out. Just last night I manually created a public domain to point to a ZeroTier address and ran the Lets Encrypt addon in Home Assistant to generate a certificate via the DNS challenge. Didn't take long, but there were many steps involved (creating a Google Cloud service account and configuring everything).

I'm very interested in Tailscale for both personal and business use-cases, but I'm rather put off by the stark centralization of offered identity providers: Microsoft, Github (Microsoft), Google, okta (?). What are the chances that Tailscale would offer authentication using decentralized/self-hosted identity providers like Ory ( https://www.ory.sh/ )?

They offer custom SSO providers using SAML or OIDC https://tailscale.com/kb/1119/sso-saml-oidc/

Unfortunately they are locked behind Enterprise pricing because of the extra help and debugging needed to get them working. Maybe at some point this will be offered standard though.

Small plug for my startup (hope that's ok!) - if anyone else is looking for an easy way to set up SSO, you should check out the WorkOS Admin Portal feature[0].

It's essentially an onboarding wizard that works for any identity provider. This makes the SAML/OIDC configuration self-serve, which in turn allows you to easily provide SSO to anyone who wants it. The UI can also be branded with your logo/colors and run on your own custom domain.

[0] https://workos.com/admin-portal

There is also the self-hostable Headscale implementation.

In case anyone is looking for the URL: https://github.com/juanfont/headscale

5k stars on Github, and lots of activity. Seems very interesting!

While not official, Tailscale themselves are not opposed to this project.

They don't much contribute to it directly, but they have gone out of their way on a few occasions to avoid breaking it or to make it easier for headscale to implement some things (like the new encryption scheme for communicating with the control server).

I imagine they don't see it as much of a business model threat, since it has no commercial support, requires having somebody your organization run and administer it, it is single tenant, not multi-tenant, so not really suitable for AWS to take and use to outcompete Tailscale proper, etc.

The people most likely to use this are the ones too concerned about security to use the official (Tailscale-hosted hosted) control plane, or people/orgs that simply cannot reasonably afford Tailscale's pricing model. In either case, they were not really customers in the first place.

obvious caveat that tailscale could, at any time, change this unofficial policy.

Note before anyone gets too excited: doesn't work on iOS, and you have to sideload an app on Android, if you want to use it from a phone.

These aren't problems for everyone but I think they should be front and center when suggesting it.

So @bradfitz when are you releasing https://tailscale.com/connect/ for real? :)

Context for the uninitiated - as a crazy idea on the podcast Security Cryptography Whatever (hosted by tptacek and others less well known on HN) Avery and Brad of tailscale imagined an ssh client in the browser with QR code authentication to SSO to allow you to connect to your tailscale network (over tailscale SSH) from untrusted computers such as internet cafes. (Or mostly untrusted - safe from keyloggers but maybe not from a dedicated active malware that injects into your browser and tries to inject secret commands into your ssh session).

I created a silly PoC here (video instead of link because don't try it for real) https://twitter.com/jgeralnik/status/1487913797155233798 back when tailscale ssh was a secret binary in the tailscale github repo

Actually pretty soon, probably.

It's working. It needs some UI love first and some docs so people know how it works and don't immediately freak out. :)

> when tailscale ssh was a secret binary in the tailscale github repo

That makes it sound like we put a binary in our git repo :) It was its own Go package main that people could run.

It was never a secret. We just didn't advertise it a ton! :)

I didn't mean binary as in "binary blob" but as in "standalone program separate from the tailscale program". Sorry for any confusion :)

I know this opinion comes up every time Tailscale is mentioned, but requiring SSO _and_ only supporting companies like Google and Microsoft on the free tier means a lot of people can't use it without being exposed to a ton of risk in the form of automated moderation/deletion decisions. I want to be excited about this stuff, but it just won't fit into my risk profile until that changes.

Hell, I'd be happy to pay $5/mo or whatever if that meant I could roll my own SSO, or even just use a cheap-per-user, low-volume provider.

This is a workaround, but could you minimize that risk by signing up for both Microsoft and Google? It should help for any policy / moderation / "computer says no" decisions that are vendor-specific.

(On the other hand, for security risks, it means that a security hole in either one would be a problem.)

That is not how Tailscale SSO works.

I can see some risk with say Google as the surface area is much larger (maybe they don’t like some play store activity or some ads activity or YouTube activity, etc). But I use my GitHub account and I’m really not worried about automated moderation locking me out of my account.

If only they allowed switching from Google to GitHub...

I'll have to ask this since it's bothering me for quite a while…

If I connect to a server via WireGuard, would it make more sense to run simpler & unencrypted `rsh` instead of `ssh`? It's kinda pointless to double encrypt.

Yeah, but e.g. no rsh (or telnet!) on macOS.

It's likewise a bit silly that we had to add TLS support to Tailscale: https://tailscale.com/blog/tls-certs/

But we want to interoperate well with the clients people already have (browsers, their system ssh client, etc...)

Is there an option to avoid double encryption on systems that do have e.g. rsh?

I might be misunderstanding the question but ... just use rsh?

I think for the same reasons I wouldn't "just use ssh" over tailscale--I don't want to have to manage an sshd that doesn't require key or password auth but listens over tailscale (and nothing else!). Basically, what I want is for tailscaled to be my rshd (appropriately configured for connections over tailscale network only, etc) or in other words to avoid double-encryption (it's not the end of the world, but ideally we don't need to doubly-encrypt).

Double (or more) encryption ends up happening a lot in larger networks not for technical reasons but for policy ones.

This is unsurprising, because it is used for different purposes in different layers of the stack. It is not at all a black and white state of "encrypted" vs. "not encrypted".

For example, in one organiztion I've worked with, Wireguard (generally, including Tailscale) is approved for restricting connections only to authorized network devices/users and that data maintains integrity in transit, but is not approved for protecting the confidentiality of sensitive information. Connections which access specific resources are required to be encrypted at the application level using a mechanism which has been approved for that information type (given a specific threat model).

So you could transmit very small amounts of data over TCP/IP, over a Tailscale network, using a set of pre-shared, one-time pads. And you might actually want to do this! It's really not ridiculous, but you do need to assess whether you really do have a threat model that needs it.

Around the time I joined Tailscale, actually just before, I had a look at rsh with an eye in this direction.

The problem is that rsh is very stale and unmaintained - even those versions that have had releases in recent years (e.g. GNU Inetutils) are very old inside - even if they've kept up with patches, they have not kept up with features e.g. modern user session construction.

It also turns out that ssh the client, much more so than ssh the protocol, is really a key integration point and API that users end up needing. It has a broad feature set that turns up in use cases all over, many of which rsh does not handle.

FWIW for those reading (I figure Brad already knows):

    echo "alias telnet=nc -v" >> ~/.zshrc && source ~/.zshrc

Alas, the real “telnet” protocol has considerably more fanciness than nc. It’s just that the telnet cli command degrades into a simple line-oriented mode if it doesn’t see the telnetd init sequence.

True, but it handles 99% of the use cases of the people who lament the demise of telnet in macOS. :-)

For the rest: brew install telnet

macports please

Here you go:

    sudo port install inetutils

Better. Thank you :)

I read that in Chris Rock's voice. :-)

For example window size negotiation

From the blog post on TLS support:

> However, if your service doesn’t have a valid TLS certificate, despite the fact that your connection is encrypted using Tailscale, your browser will warn you that the connection is not secure (it’s doing the right thing—it doesn’t know about Tailscale!). So, to avoid confusing your users, you might want to provision a TLS certificate to validate your internal services.

Browser warnings and user confusion aren’t the only consequence of not using HTTPS. The more concrete impact is that you lose access to a large and growing number of web APIs that are restricted to secure contexts.

https://developer.mozilla.org/en-US/docs/Web/Security/Secure...

Yup! In fact, that was the very first sentence of the original GitHub bug about TLS certs: https://github.com/tailscale/tailscale/issues/1235 ... "Many new web APIs (eg: geolocation, sensors, http/2, etc) require a TLS certificate"

You can still use telnet v1.9.4 on macOS. Just copy it from an old version of OSX (pre High Sierra). It still works fine on Monterey.

Yes and no. You shouldn't have rsh on your system at all -- there's a case for telnet to test connections (though netcat is better), but there's no case for rsh.

ssh used to allow setting cipher=none, but that's not available anymore.

Think of it this way: you're paying the small overhead of double encryption, but you're gaining not fatfingering your way to a password compromise.

I'm not following. How does double encryption help to avoid a password compromise if everything is authed with tailscale in the first place?

I suspect they mean that if you have rsh installed for tailscale you need to be very careful with how you run it. If you accidentally let rsh listen on 0.0.0.0 and don't firewall it then you've given attackers a way to guess passwords.

Forgetting to firewall services or accidentally exposing services to the internet is pretty common. ssh is more hardened than rsh, especially with key based auth, so the risk is lower.

Isn't this true of SSH as well? What's stopping someone from letting sshd listen on 0.0.0.0 with password auth? Anyway, I would expect that whether you're using ssh or rsh/telnet/etc that you're not configuring it for password auth, but rather using tailscale authentication. Specifically, I would expect (perhaps naively) that tailscaled handles the ssh/telnet/rsh/whatever connections rather than passing them off to another process (sshd, telnetd, etc) and thus allowing it to handle the authentication and daemon configuration (e.g., what address it listens on).

Somebody listening on local connections can sniff your password.

How does that work? First of all, I would expect that rsh-over-tailscale doesn't use password authentication, and secondly can a local user sniff traffic before it hits tailscaled?

    > would it make more sense to run simpler & unencrypted `rsh` instead of `ssh`?

No, because ssh has evolved to be so much more than "rsh with encryption".

Double encryption is twice as effective. I use double ROT-13 for double the security.

I find it more efficient to just use ROT-26

ROT-26? Hasn't that been broken for a few years now?

We all need to be shifting to ROT-52 ASAP.

Is it pointless? SSH doesn’t send the password out over the wire but instead uses a challenge-response cryptographic system so even if one of the interim machines is compromised, they don’t have access to the clear text password. You shouldn’t be raising passwords (or even passwords in the first place with ssh) but practice defense in depth.

Unless you’re transferring large files the overhead of double encryption on ssh is totally blown away by waiting for human input.

IIRC There’s a fork of SSH that supports not encrypting things if you are trying to transfer large files.

Not really. Who knows what you’re connecting to once you connect to the tailscale endpoint.

It’s more likely that you’re gonna screw up and end up doing something you don’t intend to do for very little gain. SSH overhead in 2022 is really low.

In this case, double encryption is a good idea though. Tailscale is a great way to reduce exposure of your infrastructure from the public internet, but it’s not without flaws. In theory, it should be possible for Tailscale and your SSO provider to add new nodes to your Tailnet. Though I don’t believe this is something that they’re actually willing to do, it’s definitely something to keep in mind if you’re planning on delegating SSH/sudo authentication to Tailscale.

Double encryption doesn't actually help in that case though - if tailscale (maliciously) added nodes to your network the ssh session being encrypted wouldn't change the fact that they can run commands on your machines. And if they wanted to actively MITM you they could do so (by redirecting your wireguard connection to a server owned by them) even with encryption (presuming they can fake the host key, which they could do at worst by running code on your server to steal it).

To be clear I implicitly and explicitly trust tailscale not to tamper with my networks and if your threat model includes tailscale becoming a bad actor you should remember that in that case running their binary in the first place could already be game over.

Double encryption should help as long as the Tailscale client installed on your own machines is safe. Without double encryption for SSH, Tailscale and your SSO provider can theoretically run commands on your machines without involving malicious client software. But that's not possible if you encrypt your SSH connection with your own keys.

Also for Linux, the Tailscale client is fully open source and I obtain the binary from the distro. I find that a bit reassuring.

But that's the thing - even with double encryption tailscale and your SSO can run commands on your machines

1) Run tailscale --ssh on your server 2) A malicious SSO or tailscale add a new machine to your network and update your ACL such that the new machine can connect to your server 3) ssh from the new machine to run code on your server

The fact that the connection between the malicious machine and your server is double encrypted doesn't affect the attack here at all

By double encryption, I mean using an SSH server other than "tailscale --ssh". No one except yourself can have SSH access if you disable Tailscale's builtin SSH server, use OpenSSH, and generate your own keys for authentication.

Ah, I don't think that's exactly what this thread was about. Ignoring how authorization works, the question was whether there is an advantage to encrypting your commands again (via say ssh) vs. just sending them in plaintext under wireguard (via say rsh)

Stay secure by default.

CPU overhead for encryption is basically non existend.

What happens if I use Tailscale SSH and Google (or whatever IDP) decides to ban my account? Is there a break-glass or something that would let me either change IDPs or re-enable openssh-based access without losing my servers?

Totally, I’m a happy customer, but completely relying on Tailscale for out-of-band SSH doesn’t sit right with me, especially for personal machines. One ban and poof, you can’t access your own servers.

And to be honest, I understand the appeal of not having to muck around with the system, but SSH isn’t that cumbersome once you’ve set up your /etc/hosts, and encrypt your id_rsa files. Couldn’t get any easier than `ssh <machine>`.

This bothers me as well as a personal user. I'd love to go all-in on tailscale but having companies like Google running the identity is really off-putting.

For now I think I'll be leaving the tailscale SSH functionality and keeping my own setup. I also have a static IP at home which is allowed access to my remote dedicated box. If they enabled some other identity provider that I could either self-host, or use their own with an email + password then I could close off that extra hole but for now it feels too risky for me.

On a corporate side I don't mind so much, SSO is so standard and I'd feel perfectly comfortable using Okta or Google SSO because there seem to be far fewer stories of an entire GSuite being banned with no recourse.

Tailscale is great and has solved a bunch of problems for me, but the idea of having a seperate IDP that have so many horror stories around them freaks me out enough to slightly lower my security to account for it. Thankfully there are fewer stories like that about GitHub so I'm using them for now.

Good! Boundary (https://www.boundaryproject.io/) by Hashicorp needs some healthy competition.

Teleport is also a tool in this space, for those looking for alternatives.

And for anyone looking at Tailscale, I should also mention ZeroTier (https://www.zerotier.com/).

In my opinion they have better tech, but they are pretty bad at packaging it, and bad at making it work for actual use-cases.

Tailscale seems to be much more clever around building out stuff (like this one, SSH) that actually goes all the way for a particular use-case. ZeroTier feels more like a building block, where you need to bring more stuff yourself.

Either way, both are awesome pieces of technology, and really useful!

Why is the tech better?

I had no end of problems with zerotier, connections would randomly drop between machines even when they were on the same LAN. No such problems with tailscale

Tailscale is my absolute dream networking solution, I would go as far as to say it will ultimately change how we develop applications in the future

If the auth flow was as good as Touch ID and no window switching, yeah it would be acceptable but this flow would give me a headache with all the flashing.

TouchID and related are on the list. Hooking into the existing auth flow was the easiest to get this out the door (and more desirable for some companies who want the audit event in their SSO stack, arguably), vs. figuring out when to nudge people to enroll physical factors and so forth. But I definitely also want "tap your security key please" as an option :)

But you still can't have multiple tailnets. The strategy of "have hobbyists try out the software themselves, like it, then implement it at their work" seems incompatible with this fact.

Do you use the same Google/Github/Microsoft/whatever account for both work and personal stuff?

A lot of people do just use one account for everything. Many smaller companies don’t bother giving people corporate accounts.

That sounds extremely risky. Apart from the fact that it makes it much harder to restrict access for leaving employees, mixing personal and work identities sounds like a recipe for disaster. What happens if a personal account gets banned? How do you enforce security rules?

I guess companies where there's not even any identity management, securing your network via tailscale is not your primary concern.

It's more than just a work/personal split. Even at work, having "development" and "production" tailnets so that things like testing complex ACLs, inhouse apps that use tailscale via its API, etc. are possible without having everyone on the devops team create an unmanaged/non-company email so they can create their own development tailnet, and then deploy a bunch of company IP using this rogue account.

It's a pain point.

Agreed this is a big limitation.

The only way to do it is if you have secondary email address domains. Say mdeeks@company.com and mdeeks@company.team. You can create a separate tailnet for company.team but you also have to roll out additional subnet routers (if you use them) that are authed on that second tailnet. Also you wont be able to easily write rules that interact with things that are not authed onto the second tailnet.

They need a first class concept of "canary" or "beta" that applies to ACLs, DNS configs, client versions, and all sorts of other toggles in the UI. It's a hard product problem and I'm not even sure how some of it should work.

I just know I need a way to test changes before I roll it out to everyone at the company. Right now there aren't good options for that.

I work around this issue by running multiple tailscaled daemons on different state directories and sockets.

E.g. I have the Tailscale macos application configured for the work network and then I run another tailscale daemon to connect to other home stuff:

    $ alias tailscaled
    tailscaled='sudo tailscaled --socket /Users/mkm/tmp/tailscale-mkm.socket'
    $ alias tailscale

tailscale='tailscale --socket /Users/mkm/tmp/tailscale-mkm.socket'

I installed the tailscale binaries from sources with "go install tailscale.com/cmd/tailscale{,d}@main"

This seems like the perfect complement to replace the SSM Agent / bastion instance currently used to access AWS VPC (it is super clunky to use). This should allow an easier time to do reverse tunnelling to databases without having to manage SSH keys.

Hmm AFAIK you don't need a bastion to use SSM agent - it even allows you access through the browser. I think you meant EC2 Instance Connect which manages temporary SSH keys.

You need a bastion for certain infrastructure types like AWS RDS' regardless of type (SQL Server, MYSQL) as an example. You can go direct to any EC2 hosted instance but it can get a little tedious on containers in Fargate or EC2 backed. The userspace that the Daemon runs on needs SOCKS magic to get working. Which eventually works but it's a PITA to get there and maintain.

So for our RDS instances and containers in ECR we use a bastion which IMO is a lot easier to manage.

I'm curious, what's really clunky about SSM?

Other than ensuring the pre-requisites are met, and knowing the instance-id, SSM works pretty flawlessly. You can easily write a wrapper that looks up the instance-id from the hostname, if you prefer to use it that way.

Haha I'm not sure if you were being serious, but the workflow you just outlined is the clunky part of SSM. The pre-requisites are getting all the IAM roles and permissions setup (no mean feat), installing the agent, configuring it with keys generated by another user, and getting the connection information back from the aws console. This promises to be a lot easier to setup and authenticate, install tailscale, login.

Installing the agent client side is no more or less tedious than installing the Tailscale client, IMO anyway.

I made two scripts, one in .Net with a GUI for non-devs to grep a server hostname or tag:name in AWS that resolves to an instance ID for SSH or RDP. And another python script doing the same but without the GUI for the dev team. Works a treat.

But you've already explained why it's a little tedious and now I've documented and understood why. Tailscale MagicDNS does all this nonsense for you. Yeah ok thanks for rubber ducking me I see your point now. :)

It really seems to depend on the point of view. If you're already using AWS seriously, your hosts will have the default agent anyway, IAM is already managed in a reasonable way (iamy, tf or similar), etc. so the setup is not that hard. I'm not sure what you mean by information from the AWS console - it's usable in the terminal.

I think I see what you're saying. Usually though, a lot of that stuff is single-setup. E.g., all OS's that we have deployed have the agent installed and running by default.

Additionally, the instance roles are already pre-configured.

There's almost zero overhead in ensuring SSM gets installed on new instances.

One small benefit over TailScale here, I would think, is that I don't have to rely on another tool to gain shell access. Probably a minor win, if you're running a TailScale deployment. In either case, I'd probably want to go with a single tool just to minimize the attack surface area.

That feature was recently added to SSM https://aws.amazon.com/about-aws/whats-new/2022/05/aws-syste...

Using something like gossm which I just put a PR in for this feature also makes this easier https://github.com/gjbae1212/gossm/pull/54

Oh this is awesome, I didn't see this announced. Thanks!

What would be the advantages of this compared to say Teleport ?

Teleport is working fine for us, but I wonder if the network based approach (+ wireguard) of Tailscale would be better in terms of network redundancy ?

Well, how long did it take you to set up Teleport?

It took us about an hour.

That's shocking considering my experience of weeks, nice.

We thought it was going to be a whole huge project and budgeted a week for it. It was not a whole huge project.

Teleport developer here — sorry about that. We’re aware that initial setup is at times a pain point and are prioritizing improving this. If you can provide more details about what specifically went wrong for you I’m interested in getting to the bottom of it. Feel free to reply here or you can email me isaiah@goteleport.com

Not as much as we expected frankly and each new nodes is as quick to setup as Tailscale I’d say.

The main « issue » was working with some key concepts of Teleport (logins, roles, connectors).

The big thing you get with Teleport that you don't yet get with Tailscale --- apart from entirely owning the source of truth for SSH authentication on your own infra, which is a very minor issue for almost everyone but is a major issue for some people --- is that Teleport gives you transcript-level audit logs of your SSH sessions.

Teleport also has that web-based SSH console (it's one of the better web-based consoles) and the ability to do joint SSH connections. But the audit log is the big one.

Obviously, the flip side of this is that Tailscale's SSH is built in; if you're already using Tailscale, and you're not already using Teleport, you should enable Tailscale's SSH right away; it is hugely better than managing your own SSH service ad-hoc.

Session recording's actually already in the network engine for SSH, we just haven't plumbed the whole "push recordings somewhere and surface them" yet. Soon :)

It's an extremely valuable feature, in that it can knock out a bunch of different SOC2 DRL line items with a single screenshot.

For those who are not familiar with the term DRL in "SOC2 DRL line item", it is document request list (DRL).

> is that Teleport gives you transcript-level audit logs of your SSH sessions

That is extremely valuable. Just in case 'transcript-level audit' didn't sink in, it's a session recording – not only you can see the all keystrokes typed but you can see all the outputs, the whole state. Someone doing a TOP command for an hour? You can watch the same thing later.

Think asciinema (https://asciinema.org/).

FWIW, Tailscale SSH can also record sessions in asciinema cast format:

https://github.com/tailscale/tailscale/blob/v1.26.1/ssh/tail...

We haven't yet fully "productized" it yet because it only records on-device for now. We want to make it stream recordings to another device (that you run) first before considering it done.

Nice!

Sasha, CTO@ Teleport here. Thank you for the kind words! And congrats to the Tailscale team on launching SSH product.

Let me share a bit more about our auditing capabilities:

Teleport captures session PTY output and stores it in S3 or any S3 compatible storage for your records by default.

If you would like to get additional, more in-depth insight into the session, Teleport captures syscalls, file access calls and network calls done during SSH session by correlating it with sessions' cgroup using our BPF module:

https://goteleport.com/docs/server-access/guides/bpf-session...

Teleport provides a lot of other in-depth SSH integration for auditing and compliance, for example we support moderated sessions access control with a required session moderator, or per session-MFA.

Do you own the teleport code or is it closed source?

I don't know what "own it" means, but it's open source.

I'm not entirely convinced I want a feature that adds even more exposure to the sort of goofy login flow Tailscale has.

Totally meta to this discussion: I am disturbed by the SSO/IAM trend because it gives root on the entire universe to a small collection of companies.

We are looking at a future where a security breach or misbehavior by one of a handful of companies could mass-compromise millions of businesses and critical infrastructure and possibly hundreds of millions to billions of devices. Even worse this permission is clandestine. It could be exercised against individual targets with no obvious audit trail, since auditing tends to also be delegated to the IAM provider (and nobody looks at local logs in most cases).

I guess we've been handing vendors a lot of power for a while with OS vendors that have "push" software update capability, but this is adding not only even more carte blanche permission to a small number of companies but extending it across systems running different OSes and even open source platforms like Linux and BSD.

Software update capability is also fairly coarse grained. Apple or Microsoft could push a compromised or malicious update, but it would be harder for them to specifically target a single user reliably and without being noticed. (Wait... why did I get a macOS update and none of my friends did?) SSO/IAM providers could easily do this. Imagine a subpoena that targets your SSO/IAM provider that gives the government (and maybe not even your government!) silent unlimited remote access to everything you have.

I can also imagine "cancellation" scenarios where a company doesn't like what you say so they dump your account and lock you out of all your infrastructure, requiring you to manually go around and "root" all your stuff. (If you think this would only ever be deployed against Nazis, study history a bit. Political winds shift.)

It's just a monstrous amount of power to give out, and I feel like people aren't thinking this through or maybe are not even aware of the power they are delegating.

I feel like people should at least understand what they are doing when they choose to delegate all their authentication to Google.

This is correct.

One answer is to have many, many SSO/IDP systems -- and for anyone technical enough to set up a homelab to be able to be their own IDP.

A lot of people don't let you bring your own IDP. They offer a few choices: Google, Microsoft, Okta, etc.

I can foresee this eventually being a revenue stream for a lot of companies where they charge payola to be listed as an IAM provider, sort of like the browser CA inclusion or browser default search engine list rackets.

I'd love to know of well supported, secure IDP software to use for this. I'm afraid of OpenLDAP due to its long history of security issues. What are the open source alternatives that are both minimal in configuration and solid enough to be exposed to the internet if necessary?

Keycloak is Java based, but its simple to setup and configure and requires no OpenLDAP.

It supports both SAML and OpenID Connect/OAuth2. With an LDAP backend you can also use that LDAP backend for other services that don't support those two protocols for SSO, but it is not required.

Authelia is the fast minimal solution.

Keycloak offers a much more "roll your own" design.

https://www.authelia.com/

> I feel like people should at least understand what they are doing when they choose to delegate all their authentication to Google.

I think this is my biggest concern, it's really scary to have Google Auth as literally the only barrier between no access and complete production access. I understand that a lot of the time Google accounts hold the literal keys to the kingdom anyway (customer data, internal company data, maybe source trees), but SSH was one of the last frontiers remaining.

it may also be a relic of my old age, but the idea that a browser cookie could be all you need for production root has terrified me in sso environments of past.

maybe i have an antiquated view of browser security but it seemed... unnerving.

There is a reason why many companies (especially those with few developers, artists etc, so most people run Windows), go with Azure AD as their SSO solution. Then they are only depending on Microsoft not being compromised. But they are already relying on other parts of Microsoft not being compromised, so it feels like less of a risk to those companies.

I've seen this conversation before, but I've never been clear on what exactly the consequences of the SSO are. I imagined, it might be that the provider gets an IP address when you connect or something. You're saying they potentially get _access as you_? Am I understanding that correctly?

Anything authenticated with SSO can be accessed by the SSO provider since they're able to approve any authorization, which means they can just log into all your stuff.

So e.g. if you use "log in with Google" on a web site, Google now has access to your account too (if they behaved badly or were compromised).

Spreading SSO auth everywhere gives the SSO provider login access to absolutely everything you have.

wait so if i authenticate tailscale using google and enable tailscale ssh's google can just log into any of my tailscale ssh servers?

I have not tried Tailscale SSH or looked at it deeply, but as a general rule the answer is yes if the system is using delegated SSO alone to authenticate. (What I don't know is whether TS SSH supports any secondary methods like a password or SSH auth forwarding.)

You are delegating authentication, so your delegated authenticator can authenticate anything they want.

I feel like a large number of people adopting SSO/IAM systems don't fully understand this. If they do understand and are making a cost/benefit based choice to do this that's one thing, but... I think people should understand.

I've never used or examined Tailscale either, but I assumed that:

- Tailnet traffic needs to be associated with an approved device key

- Tailnet device addition needs to be signed by the offline key of another approved device

If a compromised control plane and/or SSO provider can add and approve devices on their own then the security architecture of Tailscale would be fundamentally broken. I wouldn't even call it end-to-end encrypted.

>Wait... why did I get a macOS update and none of my friends did?

That's an easy one to circumvent if you don't need to install the malware Right Now: just wait until the next update cycle and slipstream your targeted malware in with it.

> goofy login flow

Can you be more specific about your complaints?

I edited the original to contain more detail as I posted it but it seems to have been lost somehow. The login flow for Tailscale is weird due to the need to accommodate things like a headless server being added, when combined with their use of SSO as the only method of authentication things get confused very easily.

When I add a new server I get given a URL that looks like https://login.tailscale.com/a/c44a243b to visit in a browser and authenticate the new device, the meaning of which is quickly lost as soon as you go through a Google Authenticator sign in flow, fill out some recaptchas and find your phone for a SMS token, and then the device is added to your account with no further clicks (unless you enable device authorization). It feels very weak, the connection between logging in and performing an action is fuzzy.

Due to the use of Google SSO it just has the usual problems that you get. It's not quite clear when you're logged in or not or with which of the 12 google accounts you own, it's not clear what will pop 2FA requests or login prompts. As a service tailscale has made it clear that they don't want to be an "identity provider", which means you're sort of stuck with something that doesn't feel like you can make authoritative decisions about how it acts.

SSO is not the only method of authenticating things. They have auth keys specifically for the purpose of authing headless servers. e.g. sudo tailscale up --authkey tskey-abcdef1432341818

You can also apply an ACL tag to it so that it is no longer authorized as the user and instead takes on the permissions of the tag.

In our deployments we have the headless servers pull the tagged auth key from secrets manager on boot and then just `tailscale up --authkey <value>`.

I agree the default login flow is usually not what you want for headless servers. It sort of leads you down the wrong path.

This is great -- I wish it was more plain in the admin UI that this is the better headless workflow. That seems like an easy fix!

> which of the 12 google accounts you own

Stop logging in to your personal accounts on your work machine.

It's helpful for people to know, from context later in the thread, that one of the core concerns behind this comment is the idea of using SSO at all, and thus giving "the keys to the kingdom" to Google.

Of course, it's also worth knowing that SSO is basically a universal best-practice for security teams, and while it's not de jure required by SOC2, it's almost de facto required. For once, I think the best-practices and compliance people have this one right: you are extraordinarily unlikely to get burnt for trusting Google in this instance, and the security track record of ad-hoc authentication is worse than abysmal (ad-hoc authentication is probably implicated in a plurality of all major incidents).

Probably the main concern around Google is for personal accounts where it sounds like people often find no way of recovering their accounts when Google thinks you did something bad.

For business users that isn't as much the case since there are good support options and it isn't likely you'll get locked out.

Great for businesses, potentially risky for individuals.

This seems entirely fair to me. I mean, if you're scared of Google, you can use something like Okta. If it's me choosing, it's not a hard decision: I want centralized authentication, so I can quickly do blanket interventions like enabling/disabling apps, requiring phishing-proof MFA without having to do any implementation work, and linking everything to onboarding/offboarding/access review policies. And if I'm going to trust anyone's security team with this, it's going to be Google's.

> it's also worth knowing that SSO is basically a universal best-practice for security teams

It is also best practice that some things explicitly don't sit behind SSO for defense in depth. If for example you leave CrowdStrike and JAMF behind Okta, and Okta gets popped an attacker can disable your endpoint protection and push ransomware to every machine.

Is anyone using tailscale on an organizational level?

I'm curious to hear about some of the use cases, and whether some companies and organizations are attempting to adopt this instead of traditional VPN.

I would have loved this at a company a couple of years ago which was massively all in on Google Auth for literally everything. If you're fine with that being your concrete level of authentication for everything; internal tools, external tools, etc, then tailscale sort of just slots right in, and SSH makes it even more so.

I would be very hesitant to build around this personally, hijacking Google accounts is already something pretty high value to a companies adversary, and using Tailscale and SSH like this turns it from a compromise of accounts indirectly through password resets into access to unlimited production machines, internal services, etc. It feels almost layer violating to have a soft social login through Google, that gets persisted in every Chrome browser and logged into on every employees phones also directly control SSH, but maybe that's just me.

Then host your own Auth server - https://github.com/juanfont/headscale

One of my clients, an industrial/commercial property realtor (to contextualize the environment; we’re not talking military secrets here), uses it.

Day to day I interact with it like any other VPN client except I auth via the Google workspace account they gave me.

It’s Tailscale, or hosted OpenVPN and cross your fingers they’re not snooping, or DIY Wireguard or OpenVPN and all the usual ups and downs of DIY.

Software based infra is out of the unknown unknowns era these days and years of rising usability expectations means Oracle level nightmares to deal with do not gain enough momentum to survive anymore. Tailscale is plenty easy to deal with. The only consideration is do you believe your traffic is really secure? Otherwise “it just works” like anything else these days.

That said, my project for them is deprecating the infra accessed via Tailscale (24/7 EC2 running web dashboards). The already Dockerized dashboards will run locally now and use an API to retrieve the data. Real people directly in your infra is probably best avoided.

> It’s Tailscale, or hosted OpenVPN and cross your fingers they’re not snooping...

The "cross your fingers they're not snooping" applies to Tailscale as well.

That’s what I meant.

> or DIY Wireguard or OpenVPN

What is I that YD?

> Real people directly in your infra is probably best avoided.

But I've yet to see a company where no one ever needs to ssh into a server. Using these ACLs to give a contractor access (and even visibility) to only the servers they're supposed to see is probably a big advantage over OpenVPN, where a contractor automatically becomes part of the inner network and can theoretically see all machines?

We just adopted it to consolidate multiple different OpenVPN installations.

Why?

* The Tailscale clients are dead simple and good quality (but not perfect). OpenVPN clients for mac and iOS are pretty bad. Onboarding OpenVPN users was a large document that generated a lot of questions and support issues. Tailscale onboarding is about two minutes for most users and we had nearly no support requests rolling it out widely to our company.

* Tying OpenVPN to Okta is a truly terrible experience. Users would login with their Okta creds and a push would silently go to their devices. If they didn't know to check their phone it would just fail to login. Alternatively you can paste your TOTP code after your password. Yes, really.

* We don't have to manage or debug anything related to LDAP.

* Maintenance on our side is extremely minimal. Just install subnet routers (<10 lines of bash) and put our ACLs in source control.

* We no longer have to tell users to logout and login to another VPN to get to certain resources. We just grant them access and suddenly they can reach what they need. ACLs are amazing and super easy to script, audit, and test.

* Split DNS that actually works on all operating systems. For private domain A, query this resolver (over the wireguard link), for private domain B, query this other resolver.

* I rolled it out as a PoC to all of our major VPCs in a day.

The bad? It's still a young product and is missing features and has some warts.

* Notifications on macOS that you need to relogin are just plain broken (they know and are working on it).

* We're currently battling issues with network resets due to what looks like a client bug when you have lots of users.

* No access to audit logs yet

* You can't restrict people from using exit nodes

* No good way to canary changes to your user population. Any mistake in the UI instantly breaks everyone.

> Users would login with their Okta creds and a push would silently go to their devices. If they didn't know to check their phone it would just fail to login. Alternatively you can paste your TOTP code after your password. Yes, really.

This sounds exactly like my Cisco (anyconnect) VPN experience from a previous job/life, both before and after Okta was introduced... we think it don't be like it is, but it do.

> Users would login with their Okta creds and a push would silently go to their devices. If they didn't know to check their phone it would just fail to login.

How would users not know to check their phone? They had to specifically set up this MFA method.

Because they didn't specifically set it up. That is just how Okta MFA over LDAP works: https://help.okta.com/en-us/Content/Topics/Directory/LDAP-in...

Also people just forget. Some people may only need the VPN once per month and in that time they forget about this weird login flow. They just assume they typed their password wrong or that they lost permissions to the VPN or something.

curious what 'dead simple' means re: clients. Do your users still need to login like openvpn, or is it always on?

It's a small icon in the top bar on macOS. You click login, it opens your browser, you Google/Okta auth in your browser using any factor you want (push, totp, yubikey), and you're done. Login literally takes seconds and there is little chance for confusion.

The last point is a good one, I'm not sure how that makes tailscale usable for big orgs. Imagine a company with 10k+ people using it, I guess you'd need to build a lot of own tooling to avoid breaking the whole corporate network because of a mistake in setting an ACL.

I'm somewhat more comfortable with making ACL changes because they have tests I can write in the ACLs themselves, plus I can specifically target users with new ACLs.

I'm more concerned about making any DNS changes at all. Or adding/modifying subnet routers.

I don't understand Tailscale's pricing structure: On one end, the features they are adding make the most sense if every machine that should be accessible is running tailscale.

Both the fine-grained ACL support and now this SSH thing don't make sense with shared subnets.

However, their pricing ties number of servers to number of users. In our case, we have potentially 3 admins who would administer about 50 machines, plus some ephemeral ones.

Assuming that each admin has two Macs and an iPhone just on their client side, I don't see how this can ever work within the limits in their pricing plans (except if I'd use subnet sharing, but that would cause me to miss out on many additional features that only make sense if Tailscale is running on each machine).

Is there no way to buy additional devices?

And my other gripe is with their API: The fine grained ACL support is perfect to, say, issue temporary access to some machines for some users and the API does allow that.

But why the hell are API keys only valid for 60 days? I don't want to build a solution on top of a piece of infrastructure that requires me to manually log into a site every 60 days.

1. api keys != auth keys

2. you can disable key expiry for devices where it makes sense, see https://tailscale.com/kb/1028/key-expiry/#disabling-key-expi...

1. I know. I'm talking about API keys though - they do expire after 90 days. See https://tailscale.com/kb/1101/api/

2. For API keys, expiration cannot be disabled.

Out of curiosity, what are you using the API for?

Slightly OT, How do you do "re-authenticate before connecting" (https://tailscale.com/kb/1193/tailscale-ssh/) when using Google identity, we are using Google Oauth2 and latest identity SDK but can't see how to force a user to re-authenticate if logged in, do you just make a random unique claim?

Forgive my ignorance, but what is the benefit for an individual to run this? I currently just use 1.1.1.1 by Cloudflare on my two main devices....not realloy sure I understand what the advantage of this is?

Looks interesting, but it seems that this doesn’t work well for servers where every user has a personal account. It appears that this use case would require a separate ACL entry for every user, which a) can get slightly annoying to manage and b) requires a business plan. It would be nice if something like `"users": ["autogroup:emailuser"]` was supported to allow `alice@example.com` to connect as the user `alice`, but that would probably cause issues with e.g. Github organizations, where email addresses can have different TLDs.

I started using tailscale a few days ago, and I absolutely love it.

However, one thing is still nagging me: technically, they can add devices to my network without telling me, right? Or is there something I'm missing?

Are you asking whether the owners and operators of the Tailscale control plane can theoretically add devices to your network without your authorisation? If so then yes, definitely.

Perhaps a terrible analogy, but to me the question reads like "can the bank just spend my savings?"

How might you expect a fresh node to join your existing Tailnet without Tailscale having a means to add a node?

Most code is open source, I guess they could include a feature (not enabled by default) that sends a warning whenever it sees a previously unseen device on the network. Would be noisy and useless for most, but prevent tailscale from adding a new device secretly.

But then again, I'm not sure there are many people who'd worry about that.

Requiring an administrator or other device to pre-authorize or manually approve a new device, by signing the new device key with a client signature key.

Why would you expect anything else? That’s like saying Wireguard or SSH servers should just accept any client. The purpose of mesh VPN controllers is to automate redundant key management, not to subvert the original security model.

> (SSH certificates are better, but have you tried running your own enterprise CA?)

For a small business, what is so hard about keeping a file (CA private key) secure and changing it when required?

Changing it when it is required.

> For a small business, what is so hard about keeping a file (CA private key) secure and changing it when required?

For a small business? Well, keeping a file secure and changing it when required ^^'

I mean, it's not out of this world hard to generate your private CA but there are a thousand footguns, the experience isn't exactly friendly, and it's Yet Another Thing To Do And Keep Track Off, i.e even if there's someone who has the technical chops, they may not have the bandwidth, and also, lottery factor. Let alone keeping it properly secure. There's a whole framework/procedure to create to set that up properly.

(been there done that, I was exactly in the situation above)

Can you be more specific on some of those main footguns?

I need to rotate the CA for some rare reason. Boom, I do it. All the old SSH certs are invalidated, but users can get a new one through the usual automated flow.

I love that feature but I'd be a bit scared to just switch off all other ssh, in case the tailscale service ever crashes. I know, machines can just be set up again, but if the problem reproduces there's no way to debug it.

So what's the recommendation here to stay safe but still have a failover? Keep ssh enabled for only one user (with sudo rights) and a key that's stored at some secure location?

Another question, can this be used to create SSO-enabled SFTP? Isn't SFTP just ftp over SSH?

SFTP isn't really FTP over SSH, it's just a collision of naming.

> Isn't SFTP just ftp over SSH?

FTP is a wholly different protocol, that has aged rather poorly. You might be thinking of FTPS, which is FTP with TLS. SFTP is its own thing and is actually decent/sane.

> can this be used to create SSO-enabled SFTP?

From another reply somewhere else in the comments, apparently yes.

SFTP is a sub-protocol of SSH (technically a "subsystem" in the RFC-speak), which implements features similar to "legacy" FTP.

Anyway, our ssh server knows about sftp, so `sftp <host>` should just work.

Wonderful! How would it work with gui based clients for the check prompts? Just launches the browser?

Can’t wait to try this.

This is pretty awesome! At my workplace we're using tailscale, and it's been mostly good experience. There were some hickups (like tokens expiring without sending any notification email), though all in all much better then alternatives.

You still need to manage some amount (possibly smaller than right now) of ssh keys because if not then you are totally reliant on tailscale being up all the time or you can't access your infrastructure of they have an outage.

If I have to use a browser to make use of this (which the demo shows), I never want to use it. It's like the abomination of Okta and Luminate. Absolutely horrible UX.

Nope. Will fight very hard to avoid ever having to use this.

Antagonistic toward developers at best.

Tailscalar here, you can also fetch, validate and edit your ACL's using the API https://github.com/tailscale/tailscale/blob/main/api.md#acl

my first thought is that this seems less secure than using a private ssh key and locking your machine down to only that ssh key.

you're essentially using google as your machine login, which seems like weaker security, imo.

edit: I'll caveat this and say, I think Tailscale is fantastic! I've been using it personally on my machines for a few months now, and it is awesome.

There is a reason why in a corp you need to install certain kind of ai network sniffer to get this underlying network traffic to surface. Be worked on network security and it is just hard to work in a network which you cannot see I think. The bypass is a success and it is not even free (price wise it seems). Crazy.

Tailscale is the network infrastructure for this feature. This is like being concerned that Cisco can see your ICMP headers in iOS.

Never login as root… even over secured links!

I think in this context, I'd worry about attributing any actions taken on the root account back to a named user. I suppose tailscale keeps an auth log (anyone know details?) so you likely could determine "root"="alice" by looking across different log files. Attributing privileged commands to an employee is critical for security.

In other contexts you want to avoid shared root accounts, as you'd want to block access for former employees, but you don't want to rotate credentials every time. SSO for tailscale makes that easier.

Indeed, "root" is not a person - only persons should have authorization (to log in, to elevate via su/sudo).

Ed: although in this case the binding between a system user and a person happens at the tailscale level.

"bob" UNIX account is not a person either.

If you ssh in, no matter to what account, your key ID is logged and that's what matters.

Anything can happen afterwards, unless you have a really tight grip on your system, since local privilege escallations are not that hard or uncommon.

Why? To me it doesn’t make sense not to do so. What kind of security gives you logging it as non root user and then using sudo (maybe even without a password) to become root?

Root makes everything more easier, for example to copy a file on a server with scp if you don’t have root login you first has to copy it to /tmp and the copy it in the right directory by logging in as normal user and elevating with su/sudo.

Also you have to remember which username was chosen when the server was installed, was it user, admin, or pi, or some default?

I get not using root for everyday usage on a desktop, but for a server having a non root user is not that useful.

Of course you have to know what you are doing, but sudo doesn’t protect you from stupid mistakes anyway (especially if configured with NOPASSWD as I always see doing because having to continuously type the password is annoying and password tends to be forgotten)

It probably depends a bit on your security posture and how you manage the server. The recommendation for avoiding root tends to be central to the idea of least privilege. By default, grant/allow the least privileges necessary, helps eliminate alot of security surface area, while also being a layer of protection against mistakes. It's easy to be careless with a superuser account, and have a bad day.

And this is naturally in conflict with being productive. As you mentioned, it's easier to be productive if you can just do all of the things all of the time. And operating in environments where a mistake may not be that devastating, or compromise or vulnerability this might be a reasonable tradeoff.

But I've worked in environments where this is too risky. For example: 1. Engineer accidentally pastes the wrong buffer into a terminal. They had accidentally copied some other piece of text. 2. The text happens to contain \nhostname set\n. 3. The terminal as it's spitting out errors, does see a valid command, to change the hostname. 4. That particular system was an HA system, and the process monitor in use, grepping the running processes for the command + args, of which hostname was an argument. And decided the process was no longer running. 5. The cluster seeing the failure, decides to boot another process. But at this time in history, that process was could only handle a single instance. Both instances now decided to conflict with each other. 6. Some part I don't remember about the failover site. 7. A million cell phones can no longer get an IP address.

So it's a question of tradeoffs, but is a generally recommended practice to not login directly to root, and operate with less privileges when not required. And then escalate if granted / required.

This is spot on. One interesting approach I've seen before is that all commands executed as the superuser must be written in file, and the only command accessible via sudo is "please_save_to_audit_log_then_run_it_in_sandboxed_env <file>". For particularly high-risk situations, there might be a second person reading your script before running an approval command that actually lets the script run. Things don't move quickly, but the number of mistakes via typo is certainly reduced.

I've always heard that and adhered to that, but what's the advantage of me logging in with a user account to then use sudo for every command? It's not like I could break less than being logged in as root.

Lots of reasons…

Tracking actions, fine grained policy controls, sharing a root password amongst multiple people, and then there is the risk of accidentally running a cmd as root when you thought you were in your home shell!

Confusing the server I'm on will always cause issues, not matter if I have to sudo for root.

I get the other points, but most servers I encountered don't have these fine grained controls because doing manual work on them only happens for debugging or fixing issues.

And the password doesn't need to be shared with sth like Tailscale.

Tracking actions is a good part though, I wonder if you can still track which tailscale user was responsible for that root login.

What's Tailscale? Some kind of VPN?

How Tailscale different from other VPN solutions?

They have a relatively detailed series of comparisons with other solutions in their documentation: https://tailscale.com/kb/comparisons/vpns/

Reply on news.ycombinator.com