About the Tailscale.com outage on March 7, 2024
tailscale.com
tailscale.com
I still marvel at just how good Tailscale is. I'm a minor user really but I have two sites that I use tailscale to access: a couple of on-prem servers and my AWS production setup.
I can literally work from anywhere - had an issue over the weekend where I was trying to deploy an ECS container but the local wifi was so slow that the deploy kept timing out.
I simply SSH'd over to my on-prem development machine, did a git pull of the latest code and did the deploy from there. All while remaining secure with no open ports at all on my on-prem system and none in AWS. Can even do testing against the production Aurora database without any open ports on it, simply run a tailscale agent in AWS on a nano sized EC2.
Got another developer you need to give access to your network to? Tailscale makes that trivial (as it does revoking them).
Yeah, for that deployment I could just make a GitHub action or something and avoid the perils of terrible internet, but for this I like to do it manually and Tailscale lets me do just that.
I use mosh and gnu screen for flaky connections. Works wonders even if you disconnect every 10 seconds.
I'd recommend as part of the post mortem to move their install script off their marketing site or putting in some other fallback so marketing site activity is unrelated to customer operations critical path. They're almost there for maintaining that typical isolation, which helps bc this kind of thing is common.
We track uptime of our various providers, and seeing bits like the GitHub or Zendesk sites go down is more common than we expected... and they're the good cases.
When we picked that URL, the marketing site was created and run by the same people who built the rest of the product, so it didn't seem like a concern at the time.
(In Google Cloud you could do it entirely with load balancing rules, no need to even run a web server)
From an external perspective no one cares if www going down isn't "your fault" or of "direct impact to the product". It's a corporate blackeye either way.
It's impossible to know because they won't admit it publicly. You are guessing based on some anecdotal experience.
But then again... here's mine! I worked at a very successful SaaS that had (really not kidding) the most incompetent, lazy dope running the www site. He live-edited a "staging" version of the site on the fly (no, it wasn't private, you could access this thing from the internet, and he didn't know or care about that). When he was happy with his changes he'd destroy the live instances behind the load balancer and clone his staging instance without taking it down or running any extra checks. This staging instance was around for years and I don't think he ever bothered doing a system update. Since he didn't use git, I I'll bet that at least once he cloned a live instance back to staging to undo a bunch of bork.
I lost count of the incidents. He never detected them himself, was never available to troubleshoot them and was generally a big "durrrr" when you'd finally get him on the call. Example: one time we had a "slow, intermittent errors" customer support ticket surfaced to us, not because it was our job, but because dopey was being an absolute ass to the helpdesk guys. He ran his crap in another AWS account we didn't have access to. About a day later the www site went down completely, so we got hold of the AWS account and dug in. All 5 of the instances behind the load balance were "unhealthy" for various reasons. Certs expired, disks full, apache stopped. We bounced them, restarted them and sshed in. They all had different versions of the site. It was a complete mess. Turns out dopey wasn't very good at killing the old instances and cloning staging. He was probably live-editing the instances for smaller changes if that seemed easier than a bunch of AWS console work.
Unbelievably he wasn't fired and continued to mismanage the site, and we could do nothing because the head of marketing didn't listen to the head of engineering. They hated each other. The way Marketing saw it "your SRE guys couldn't fix it, they had to wait for <dopey> to get on the call". I'm not even kidding.
Just more anecdotal evidence from me. You might be right.
Reminds me of the time marketing decided to change the logo on the marketing site for the product team I was on without being aware that the site was scraped and redeployed on a different domain (by hand). When the logo changed, the CSS for the image element wasn't updated, truncating part of the logo, proudly displaying the word "ass" as a part of the logo in an unfortunate cropping incident.
† A systems architecture tightly coupled to the structure of the organization in which it was created.
`curl -fsSL https://install.tailscale.com | sh` wouldn't be any less nice. Append /sh if having something human-friendly at the root is desirable (SEO, etc.), and you're still at the same overall length as today.
Even this isn't really necessary; curl includes a default user agent header identifying the traffic coming from curl. It's simple enough to direct traffic with the curl user agent header to the script and all other traffic to a static website with directions for how to quick-install.
* The user wants to read the script before executing it, and their preferred reader (perhaps due to browser extension or something) is a standard browser.
* The user has `curl` aliased to `curl-impersonate` in order to avoid things like Cloudflare's bot detection (a captcha that triggers on things beyond the HTTP request, like the less fancy TLS handshake of regular curl) -- https://github.com/lwthiker/curl-impersonate
* The user doesn't have curl installed, but has wget / lynx / some headless browser / etc. and expects any of those to work the same as curl.
Not to mention, if a site encouraged users to execute an HTTP response by piping curl into sh, and the response for curl was different than the response otherwise, that just might make the top of HN for being sketchy as hell.
I mean, the point of wanting to read the script before executing it is to try and protect yourself from malicious scripts that abuse the curl | sh pattern. So since it would be simple enough for a malicious actor to return something different when the user agent indicates the usage of curl, the only responsible thing to do, anyway, is to use curl to download the script to a file, read the file, then execute the file.
> `curl` aliased to `curl-impersonate`
So when the user uses a tool to impersonate a browser, they'll see exactly what they'll see in a browser... which are the quick-install instructions anyway, which can include a note about the user agent, if anyone actually hits this in the real world?
> wget / lynx / some headless browser
Which would provide the quick-install instructions to use curl :)
Cloudflare seems to handle a fair bit of this if you host your domain with them, but you have to use Cloudflare.
It wasn’t until our first marketing web site outage that we realised that our $40/mo hosting plan was not merely hosting a “marketing site” but rather critical infrastructure. That was a load-bearing $40 hosting plan. Our app wasn’t down but the users thought it was.
I learned then that users follow the trails you make for them without realising there are others, and if you take one away then a segment of your user base be completely lost.
My browser used to autofill dash.cloudflare.com when I typed in cloudflare. I visited the cloudflare.com website exactly once and now that's what shows up in the first result, and I find myself doing the same thing with Cloudflare.
www for the website
app, signin, login, entr, for the sign in page.
The basics don't cost $18/user/month though. The whole package does. I hear what you're saying, and maybe you just accidentally worded it this way, but the obvious rebuttal to it is: How much would it cost you to set up a solution where only ACL'd users can SSH into your infrastructure/servers? You're looking at services that cost money like Userify for that. For many of the other features Tailscale offers, you're probably either paying another service to handle that responsibility, not doing it at all, or you're spending your time recreating it, and I bet your time isn't cheap to the company either.
Anyway, that's somewhat of a hypothetical rebuttal. I actually assume you did the due diligence and weighed the cost with the portion of the feature set that you actually would make use of. I could see the price being more fair if they offered a lower cost tier that only provided the VPN and ACLs for unlimited users, but I'm not a savvy businessman so I'm not sure if it makes sense for a multi-tool company to sell a minority of users a screwdriver.
So yes, maybe a small subset of my users are actually using enough of the premium bundle to justify that cost but I can't even mix and match because the basis of every use case (the ACLs) are only in the premium package in a functional way.
“Im not going to use most of your features, can I have 75% off” isn’t as outlandish as it sounds. Willing to bet they would bite.
The pricing isn’t ridiculous, it is by design. For better of worse SaaS pricing is about finding features (regardless of their actual cost) that act as signals for “is a customer who can afford more”. The $6 tier was you paying their marketing and market research cost by you trying them out :-). They probably don’t need the $6 they need the data that you were willing to pay something!
Tailscale has competitors too with some overlaps, it might not be fully what you're looking for.
All I know is within a few minutes I had more of a project working together than without it.
It really is one of the more remarkably simple tools out there for everything it does, and has a generous free tier with 100 devices and 3 users.
Configuring wireguard really is that hard. Tailscale is easily worth it
Considering it can also run in a docker container, it’s next to trivial to install locally to try out
it’d be interesting to know why, I use it frequently at work and it’s worked pretty well so far.
But, if you are going to self-host, seriously consider Nebula instead of tailscale. Unless you need non-technical users accessing it, tailscale has a better story there.
(edit) The biggest downside of headscale is I don't feel confident I can update ACLs without having a high likelihood of taking down the entire tailnet until I can get it fixed.
Although I admit that in my role I have quite a lot of weight in convincing management on these topics, price was not a concern.
We’ve been a happy customer since April last year, everyone on the premium / “expensive” tier. I’m also very impressed with their development speed: some features that were said “May take a few years to be delivered” actually were delivered last year already.
Cloudflare One could have been an alternative, but that would have been even more expensive.
Basically, tailscale has a bundling problem. They bundled necessary to all features (proper acls) with a bunch of premium stuff that is of less value to many and they don't have the market power of microsoft with their windows operating system to force that kind of arrangement down my throat. They need a 2-3 dollar a month tier with proper acls and mesh vpn, then al a carte of the rest of the feature bloat per user (ssh key management is worth no more than 2 dollars a month based on the competition, no idea what the other features are worth because I have no use for them).
They also really need to improve their windows experience. More than once during testing I had a windows update break the vpn requiring alternate means of logging in and reconnecting, but that's an ancillary issue.
$ host www.tailscale.com
www.tailscale.com has address 76.76.21.21 # Vercel
www.tailscale.com has IPv6 address 2600:9000:a51d:27c1:6748:d035:a989:fb3c # Amazon
www.tailscale.com has IPv6 address 2600:9000:a602:b1e6:5b89:50a1:7cf7:67b8 # Amazon
> We are targeting to land support for IPv6 towards the beginning of next year. We will communicate updates on this issue.
Was from 2023-10-01, I guess it's early until June 30.
https://github.com/orgs/vercel/discussions/47#discussioncomm...
It painful to see tech providers go down this road, which is pretty similar to what's happening at Boeing. (Business taking over Engineering)
https://web.archive.org/web/20240221195021/https://github.co...
>I apologize for the slow response. We are targeting to land support for IPv6 towards the beginning of next year. We will communicate updates on this issue. Thanks for the patience.
So cringey. Why not just post a new post that said "sorry the deadline slipped, no new date available at the moment"? I will strongly recommend _against_ this company solely based on this communication. If this sort of gaslighting is how they handle their public comms, imagine how their support must be run.
That being said, still some unanswered questions:
- If the issue was ipv6 configuration breaking automated cert renewals for ipv4, wouldn't they have hit this like.. a long time ago? Did I miss something here?
- Why did this take 90 minutes to resolve? I know it's like a blog post and not a real post-mortem, but some kind of timeline would have been nice to include in the post.
- Why not move to DNS provider that natively supports ipv6s?
Also I'm curious if it's worth the overhead to have a dedicated domain for scripts/packages? Do other folks do this? (excluding third-parties like package repositories).
AIUI, they switched to their current setup 90 days prior to the outage. The initial cert they installed during their migration lasted 90 days. So 90 days after the migration, they had an outage.
Heck, a TCP proxy might even allow automatic renewal to work if the domain validation is being done using a TLS-ALPN challenge.
QUIC, in particular, is harder to proxy (if you're load balancing, say: https://quicwg.org/ops-drafts/draft-ietf-quic-manageability....).
If it is point-to-point and you control both those points (forward A to B with ports open as approp), proxying any protocol should be straightforward, no?
Hmm, it looks like Tailscale is using NetActuate for pkgs.tailscale.com. I bet NetActuate could help serve up a non-terminating proxy with plenty of PoPs at a reasonable price. Their website doesn’t give pricing, but it sounds like the kind of company that doesn’t mark up egress 50x.
Are you really getting any latency or availability improvements in that case? What does a non-TLS-terminating proxy give you?
The original proxy was stood up quickly when it was first discovered IPv6 was broken and the people standing up the proxy didn't know at the time how ACME worked.
We'll be changing it to just a TCP proxy.
yikes
This isn't a problem if you don't need the user's IP address at all, but it's often useful for logging and abuse detection.
[1] https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
We need a better story on this.
Edit: this is the only necessary fix, no need for calendar invites:
> We also plan to update our prober infrastructure to check IPv4 and IPv6 endpoints separately.
So 90 days of alerts about certs, and then certs fail?
Love that line
2~ hours into my search, contemplating building my own, someone pointed out we can just use a shared gsuite calendar.
How the mind overcomplicates things sometimes..
Part of my love/hate relationship with JIRA was until the lightbulb that it's not supposed to work perfect out of the box because no two places are the same.
Devops is so 2023. Back to ops!
(But super short term, yes.)
A simple cronjob would look like it would handle it, but what usually ends up being needed with 10-15 of these types of tasks is a simple, independent bpm workflow platform that tracks whether it happened or not.. or anything else.
Learned this the hard way and won't do it any other way.
The choice of IPV4 + shenanigans vs IPV6 seems pretty straightforward.
Certificate Transparency is used to account for maliciously or mistakenly issued certificates. Perhaps it could also be used to assert the unavailability of correctly issued but obsolete certificates that are believed to be purged but actually aren't. (Services like KeyChest might already do this.)
Let's Encrypt is a miracle compared to the expensive pain of getting a cert 20 years ago. Rather than resting on laurels, would there be any benefit to renewing even more frequently, like daily? This might have confined the Tailscale incident to a quick "oops!" while the provider migration was still underway and being actively watched.