CVE-2022-41924 – tailscaled can be used to remotely execute code on Windows
emily.id.au
emily.id.au
ps. she's looking an employer rn // hire her!
There's a lot of randomness in which submissions get noticed and achieve liftoff from /newest. At least it evens out over time if you continue to post good submissions!
Then again it does seem like the entire universe applies "eh probably nobody will try and hack it" to services listening on local TCP interfaces.
They certainly don't care about multi-user machines, though I suppose there are so many local root exploits these days you're basically trusting your users anyway in that situation.
Linux is used by by docent "security companies" and still has vulnerabilities from time to time. So does Apple in the products where their care about security, etc.
More important are three things to realize:
1. their handling of the incident, which wasn't just fast but you could say absurdly fast to a point that I'm pretty sure multiple employees dropped everything the moment they read the mail to solely focused on fixing and analyzing it
2. it's Windows only (at least it's main problem is), coming from a workaround for a feature "missing" in windows from a company which is relative young and started out in the Linux/UNIX space.
3. it is exploitable due to a fundamental design flaw of browsers
Or what I'm trying to say: It isn't that surprising (or worrying) that such a thing happened, what matters is how they handle it and make sure that it will not happen again.
It's not missing. Windows has named pipes. They just don't use the same API as Unix sockets so you would have to do some work to do it properly (or use an existing library that abstracts both interfaces).
In any case Windows has had support for Unix sockets since 2017 so there's really no excuse.
> Linux is used by by docent "security companies" and still has vulnerabilities from time to time. So does Apple in the products where their care about security, etc.
Right but neither of them are companies whose sole product is a security product.
I agree their response time is good - they probably realised how bad it looks!
“None of these words are in the Bible.
This is certainly true of the King James Version, but many of these words can be found in what might as well be the Old Testament of Same-Origin Policy: the WhatWG Fetch Standard, which defines the CORS rules we are being accused of violating.”
Now I feel less crazy for not using Tailscale SSH for similar reasons.
I'd like to see a security evaluation of Tailscale, on a per feature basis.
I'd like to see tailscaled run with far fewer privileges.
Is there a Tailscale alternative that just does Wireguard + NAT traversal and doesn't try to do key management?
Unfortunately reading about this remote RCE vector has me wondering whether I can use the product at all without all this bloat (taildrop, ssh, etc) affecting me. Going to have my team look at zerotier this week, I’ve heard a few ok things.
"Zerotier: multiple vulnerabilities lead to private network access."
The zerotier software failed - as such you could (in the simplest terms) bypass the transport “firewall”. At no point could you execute code on my machines. At no point could you spoof any authorization layers outside of what’s required to reach my ports. So when the model catastrophically failed here, attackers still cannot login to my machine. Other attacks might make this possible (e.g. code exec in the agent), but were not found - I suspect due to the lack of attack surface.
Outside narrow very well defined cases where proofs of security are possible, it might be impossible create perfectly secure computing systems due to the insolubility of the halting problem and the sheer size of the combinatorial space.
If you watch the CVE announcements it's a continuous stream of serious bugs in all kinds of major software applications including OSes, web browsers, networking hardware, VPNs, cryptographic libraries, and so on. Microsoft, Apple, Cisco, etc. have serious vulnerabilities fairly often.
This vulnerability is very critical, and discovered by an undergrad (not a security team): Code execution in local machine, taking over tailscaled, hijacking the coordination server, adding nodes, SSHing into machines, SMB shares, etc. The users are owned if attacked, and this was supposed to be a security-focused product.
Part of the problem is the feature bloat, that Wireguard deliberately avoided. Like, I want a mesh VPN, not an alternative to OpenSSH or Dropbox as well. The integrations add code, and it’s hard to secure a larger code base.
The response from Tailscale has been excellent though. Hopefully they will take measures to prevent such issues. This is a VPN after all!
True, but even the bare minimum WireGuard VPN still has a lot of stuff other than WireGuard. There's going to be a configuration protocol, software to create a tunnel device on the system, a management protocol, software updates, a UI, identity management or some kind of login/auth system, etc.
Now imagine you are an enterprise user of tailscale, you diligently elected not to trust it with login to your boxes, but you still got pwned because of “taildrop”, a feature no one on your team uses, wants, or knew was enabled.
Software vulnerabilities happen at a rate that highly correlates with size of attack surface. The attack surface here is pretty clearly too high (bad “defense in depth” as you say), and I hope they provide mechanisms in the future for disabling all this bloat, otherwise offerings like zerotier will eat their lunch.
Yeah - I have a dislike for services running as root when it's not necessary, and then getting users to escalate to root to interact with them routinely.
One thing I was thinking about was trying to identify the Linux capabilities which let tailscaled run, and then look at if it's feasible to adjust the default systemd unit to run it as a non root user. Closely followed by then trying to harden up the service with as many of the recommendations as possible in "systemd-analyze security".
Despite there being a pretty good range of restrictions available, it seems to be pretty rare that service definitions actually come locked down... Might be something for the tailscale team to look at in future?
If so, the config is straightforward for techies, just Wireguard config, it routes into my home server and I use Apache Reverse Proxy to route to the backend services.
More info on how it works: https://www.jordanwhited.com/posts/wireguard-endpoint-discov...
I guess it is because it's easy ? But now even windows can make unix sockets, that seems like reasonably easy and secure solution for "talk with some daemon portably"
I really wish there was a NAT traversal protocol or library that wasn't overly complex and focused on the 90% cases. It would help not just tailscale's but anyone building p2p tech.
Anyway, it would be much better to leave the socket APIs to handle this, possibly with OS safeguards and privileges. Writing p2p applications is analogous to being constantly protected "for your own good" by a guardian. /rant
Much less relevant when now even windows comes with half decent, reasonable default firewall out of the box. Then again "user clicking allow button till it works" is still a problem.
Who'd pay for the routing? :)
I don't think it offers authn/authz, but that's fine: neither does my ISP. I just want SSH reachability.
Focus is more on device identity/posture, DNS + remote access rather than straight VPN like Tailscale & co
Would be nice to get a blog post from them that goes a bit into impact, not just a report that tells you to update. It's nice that they responded quickly, but I feel like this shouldn't have happened in the first place for a network security company and it makes the Windows client feel like a bit of an afterthought. Looks like they have a PR open to switch it to named pipes, I hope that is properly reviewed by someone that knows Windows APIs before it's merged.
> Am I affected?
> Yes. Your tailnet has at least one Windows node running a version of Tailscale prior to v1.32.3.
Original comment:
> Do they have enough logs to reach out to people that were affected?
It happens on the client, there are no server logs that Tailscale could check
My guess is the client sends some kind of "goodbye" message when it gets reconfigured to another coordination server, and that message has enough information to determine if it originated from this attack.
It's not that bad actually.
Does anyone know why? It seems like an obviously good thing to have.
But even if browsers now implement PNA the tailnet itself is public address space, so that vector still exists. I wonder if browsers (and eventually standards) will be pressured to treat those blocks as private.
If your private net is full of trivial to access things with no access control or horribly insecure services, that's a huge problem. There are many many many ways to hop over firewalls. Hostile JS on web sites is just one.
Network boundaries are only first lines of defense in what should be a defense in depth strategy. Never depend on any one single boundary completely.
My personal criteria is: if it's not secure enough to be connected directly to the Internet with no firewall, it's broken. Make it that secure and then put it on a secure network.
The lack of integrated authentication services is one reason why so many things are completely open. It's too hard to set up and manage user credentials, and in any case, programs shouldn't have access to user credentials, they should get delegated permission. AD has made everything too hard, we need a TOFU like dynamic machine identity exchange, which then allows individuals users to execute programs with particular network capabilities.
In my opinion, this should be done not only for non-HTTPS services, but for all services: the "default" virtual host (used where there is no Host header, or when it has an unexpected value) should have nothing except a static 4xx error page. This not only avoids DNS rebinding attacks, but also avoids automated attacks in which the attacker doesn't know the correct hostname for the service (mostly automated scans for vulnerable PHP scripts and similar).
Windows has APIs (named pipes, DCOM (eww) and such) that allow authenticated local access to services. Unixes have unix sockets.
They actually approximate this functionality in the Windows implementation: It checks netstat to enforce that incoming TCP connections are from the expected Windows user! https://github.com/tailscale/tailscale/blob/2a991a3541ae5d56...
That's why we were happy with the solution they implemented as a stopgap, until they could switch to named pipes (which there is now an open PR for).
It feels like there could still be a TOCTOU issue there, but it'd be difficult to use.
All the major cloud get this IMO entirely wrong with their services that issue secrets to instances (e.g. AWS IDMS).
(I don't know anything about Tailscale so I'm just going on first principles.)
Tailscale client downloads are extremely slow at the moment, so I suggest you distribute one copy manually around your tailnet rather than bogging down their servers even more.
The Windows client caches the current version for a while, so may not yet have v1.32.3 available on your device. In that case, you can still pull the latest release from http://pkgs.tailscale.com/stable.
Or Get Tailscale in the Windows Store so it could auto update for all the endpoints out in the wild I don't control (company laptops, users home PCs, etc). Trying to get TEN people to manually update today was a pain, I can't imagine even triple that.
Who is affected?
All Windows clients prior to version v1.32.3 are affected...an attacker-controlled website visited by the node..rebinds DNS for the peer API to an attacker-controlled DNS server making peer API requests in the client, including accessing the node’s Tailscale environment variables
I have mixed feelings here as a Tailscale customer.
Yes a quick response is great, but this actual security issue is pretty terrible IMHO.
Anything other than an immediate response would have been akin to lighting their company on fire and walking away.
Have we forgotten Zoom, who reinstalled itself secretly on user machines with an RCE-vulnerable server, which they described as “working as intended?” They’re still wildly popular today with organizations despite the insane lack of regard for security and their users’ safety.
Mistakes happen. I applaud Tailscale for moving so quickly and doing the right thing.
[1]: https://notes.acuteaura.net/posts/github-enterprise-security...
Companies get away with sweeping customer security issues under the rug less and less, but still far too often. I honestly wish we as a people would put other players of this game in as high standards as you do here for this company here.
Wow, I used to think Linux security was miles ahead of Windows security more than 20 years ago because of insanity like this. Fast forward 20 years. NTLMv2 is common, so cracking a password actually requires guessing the entire password instead of just 8 characters. But password guesses are much cheaper, so we haven’t gained much.
Microsoft, how long will it take you to fix this for real? Opening a URL or UNC path should not, without an opt-in, authenticate at all. If configured to authenticate, it should prove, zero-knowledge, to the server that the supplied password (e.g. the logged-in user password) matches the server’s expected password. No further information should be leaked.
The speed and quality of Tailscale's response to our report is unlike any vendor interaction I have experienced, and suggests a deep commitment to keeping their customers safe."
Every product has security issues now and then. The real challeng is building robust processes remediate them (and to ensure that class of issue doesn't reoccur). The teams that deliver, by being transparent and fixing their stuff in a timely way, get my business.
It is dead easy to export a vulnerability scan or penetration test report and throw it at the developers, but you will get much better outcomes and better rapport if you tell them what they need to do (i.e. patch to version x.x.x) versus telling them what is wrong ("the sky is falling!").
> Sat 19 November: Coordinated Disclosure time proposed by Tailscale, accepted by us, Tailscale shares planned Security Bulletins and blog post > Tue 22 November, 5:06AM: Blog draft shared with Tailscale (a bit last minute, sorry!!!) > Tue 22 November, 7:00AM: Coordinated disclosure time
Because the code is open source anyway, I'm guessing they assume attackers would see the announcement of a vulnerability, browse the recent pull requests and figure it all out themselves anyway. Delaying publishing of the details saves maybe a few days of exposure to risk for motivated attackers, especially as the author seems to have done her work together with one other person in just over a week.
They've also sent out emails it seems, so people know they should update ASAP and why. With the extremely limited amount of people running Tailscale (and the even smaller subgroup running it on Windows specifically) I don't think it's an attack hackers will rush to roll out. Mitigations also exist (i.e. block access from the browser to 100.100.100.100) so even in situations where you cannot update you can protect yourself.
> Coordinated Disclosure time proposed by Tailscale, accepted by us, Tailscale shares planned Security Bulletins and blog postThis is why I disable javascript by default, but I suspect that on this page it's needed to fix the theme or something, because the text is light grey on a white background, and all monospace sections are completely illegible.
Edit: I don't mean to hate on the author, the content of the article is really interesting!
You seem to be correct. I found a single <script> tag in the source, with the following code:
(() => {
let v = localStorage.getItem("color-scheme"),
a = window.matchMedia("(prefers-color-scheme: dark)").matches,
cl = document.documentElement.classList,
setColorScheme = v => (!v || v === "auto" ? a : v === "dark") ? cl.add("dark") : cl.remove("dark");
setColorScheme(v);
window.setColorScheme = v => {
setColorScheme(v);
localStorage.setItem("color-scheme", v)
};
})();
Looks fine if I inspect and set the bg color to black so I think author just didn't test dark theme properly...
> This is a pretty bad idea, but luckily even the web browser has its limits.
Vulnerabilities are inevitable, the actions taken in the hours (ideally) and days following the discovery is what matters most.