Google Kubernetes Engine incident spanning 9 days
status.cloud.google.com
status.cloud.google.com
What chance a bunch of poor sys admins stand running bunch of k8s clusters for a mid size company I wonder.
Every time I think of deploying (self managed, have done full stack) it for something mission critical, this upgrade scenario simply makes me rethink it altogether.
And even managed k8s has no guarantees and if managed is to be the option, nothing beats ECS in simplicity and smooth operation at certain scales.
PS: Full stack K8s means ingress controllers, DNS auto registration, GitOps , logging, monitoring, CI/CD and all the bells and wistles including a management UI behind oauth etc.
Sometimes a clean install of fresh cluster and switch over is so much simpler if we can just take a few minutes downtime to cut over.
Complex requirements (sometimes) end up with complex solutions.
It would be good to think of the occasional downtime during a significant upgrade (once in 6 months?) separate from the ability to automatically replace failed nodes, scale up and down, having all the k8s magic.
there are other variations of this, but this seems the most obvious. blue/green deploys and the like.
Booting up a cluster is fine but the real struggle starts from that point onwards.
K8s is a great example of a "tech jobs program": technology that is absurdly complicated to the point that you have to pay someone lots of money to make it keep working, but you can't afford not to pay for this, because everyone has decided to be stuck on it and now there's no alternative (and probably never will be)
No, it is not. K8s is an example of how it's impossible to be a jack of all trades and retain simplicity. If a tool seeks to be all things to everyone, then there's no sweeping the inherent complexity under the carpet.
Full-coverage solutions to complex domains are themselves complex by necessity - see SAP, Salesforce, Oracle, Windows backwards compatibility, or try writing a parser/stemmer for a spoken language that evolved naturally :)
You can make something that has complexity without making it complicated or expensive to maintain. Take cars for example. There are many cars today that have a barrage of complex features, yet don't have costly maintenance bills and aren't difficult to use. That's because they have been engineered purposefully to reduce that complexity, and because the people that make them are masters of their craft, trying to create a premium product. Materials engineers, structural engineers, sound engineers, mechanical engineers, and more, all work toward the best design possible, to implement the most functionality with the minimum number of defects. They can tell you how likely it is for the paint to begin chipping on a new tailgate due to potential contamination in an alloy. And they'll use that knowledge to make it less likely that you'll have to take your car in for repair.
The software world today - especially where customers are an afterthought (such as backend software) - does not make attempts to improve its design or implementation to reduce the need for repairs. Instead they just "manage" it for you, doing constant maintenance on the constantly breaking vehicle they drive for you, while charging you a subscription fee. (the auto industry noticed how profitable this is, and is now doing the same wherever it can)
K8s is a great example of terrible design and half-assed implementation. It was not designed to be easy to install, configure, operate, upgrade, or maintain. It was designed the way a company designs software for its own internal use: start with some "opinions" that are more about inflated intellectual ego than practicality, throw some shit together that "basically works", and keep introducing breaking changes so it can never be operated continuously without requiring constant reworking and changing how it's run.
If a car were produced like this, nobody would buy it. But it's handed out "for free", so we all just accept it and end up driving this janky-ass school bus that constantly needs its components changed while the bus is running. But you can't ask a regular person to change the parts on a bus mid-cross-country-tour, so you have to hire a mechanic to sit there on the bus. It doesn't have to be that way, it just is because the people who made that bus don't give a shit about hiring a mechanic, they have 10,000 mechanics already.
IF K8s were a vehicle, it wouldn't be a car; it would be a multipurpose monster that could be used by some for getting groceries, or configurable into an 18-wheeler for interstate trucking, or as a off-road missile launch platform and somehow also amphibious/submersible. That.... thing will be complex because it has to be able to achieve a variety of tasks.
Also, cars are still complex beasts that have achieved economies of scale (I've had "repairs" where they just replaced an entire assembly instead of fixing a smaller component that was broken). I'm sure there are repair forums were mechanics complain about cars more than HN complains about K8s.
> K8s is a great example of terrible design and half-assed implementation
So where are the superior competitors by the master craftsmen with good taste, then? People willingly choose Kubernetes because the alternatives (third-party, homegrown or manual processes) are worse. Ops, configuration and orchestration management has been steadily getting better over the last few decades, but there is no silver bullet. If you know of one, I'd be happy to try it out and save a lot of money.
The competitors don't exist because incumbency prevents progress. Nobody wants to write a competitor to K8s, because it would be impossible to get people to switch away from K8s, because K8s is a billion-dollar ecosystem. Thousands of companies now exist solely to service it and build parts for it. None of them want to support two such systems, and none of them want to completely reinvent themselves for some other platform. And from the user's perspective, it's not like getting into a different car, it's like learning to drive an entirely different vehicle, so they have no incentive to switch.
This isn't just a tech thing, this is a universal thing. Incumbents suppress innovation and competition without even trying to.
Except, we haven't seen any of these nimble players gaining any traction. Maybe they're out there, or maybe they're yet to be created. Or maybe the complexity is an integral part of the problem they're supposed to be solving.
I guess we'll know in a few years.
Angular was an incumbent and still got toppled - albeit by a project from another tech giant (React). The question is - why hasn't Microsoft or Amazon incubated superior, simpler alternatives to K8s that works best on their respective clouds? The financial incentives are clear to me, but there's a big hole where that product ought to be
It is a blackbox with many competing implementations and each implementation even differing in implementation details within it's own versions.
In theory, it seems reasonable to say that K8s networking is an implemention detail which you shouldn't bother about but in practice, it's going to bite you because of some misconfigured component somewhere someday and you would have no idea what to make of thousands of iptable rules or BGP routes or even least inspectable eBPF mini programs.
Paid support, consulting.
Most application developers don’t give a rats ass about how 2 nodes are connected as long as the pods can speak to each other. Throw these guys in a devops team and give them the task to setup a 2 node k8s cluster and guess how much consideration they will pay to networking. Correct, zero. Except you can’t, because you have to choose a CNI plugin or choose a distribution that has made an opinionated choice for you.
To lower the barrier of entry. A default installation must become a lot more opinionated with much less options and more sane and safe default.
I chose Nomad for my current role and it works fantastic.
I used ECS in my last role and it was also fantastic.
I'm currently gently looking for a job and the market is brutal. I'm not even getting responses back from applications and I have ten years of experience. Every job posting I see requires deep k8s knowledge and hands-on experience with some ancillary k8s project like ArgoCD or Flux. I'm actually thinking about downplaying non-k8s experience on my resume and just floating the k8s that I do have straight to the top because, at the very least, it'll get my resume looked at.
If I was going to go back in time, I'd pick k8s instead of Nomad or ECS even if they weren't the best contextual choices because, hey, I gotta' put food on the table.
I wish cloud providers would just communicate outages to services I use like this to me!
Slack let’s you subscribe directly to an RSS feed using /feed.
Sounds like a good approach nonetheless
Not once has a status page that's devoid of information or slow to update ever saved face. I am far more annoyed to have to continue to verify "no, it is indeed that your service is down, not mine" and then file a support ticket. I am triply annoyed if the response from support is "ah yes that's a known problem and we're working on it" — known, and you just didn't bother to communicate.
I miss the days when Github had graphs. Even if they simply hadn't had the time to put a message on the page, you could tell from the graphs that it was Github. But even with "more information" that some PM might not like being put out publicly, Github felt more reliable & stable in those days.
At the end of the day, no amount of political gamesmanship will save you from having to actually run a reliable service, and gamesmanship just makes it more likely I'll ascribe false positives to your service, further lowering my perception of its reliability.
It's so watered down that the "AWS" emoji in our Slack instance is literally a meme of the status page.
I am sure that counts. Probably tied into someone's bonus as well.
very likely not just one.
not sure how this could work, as some qa/sre would have to be paid even more in order to effectively work against this type of falsification. there would have to be a very strong incentive for the company to do that.
See all those green there? Once it started becoming "monitored" by VPs instead of the software engineers on call, they started to become political. I bet there are several sev2s happening for several of those services even as we speak but it still shows green to the outside observer. If one has access to the actual metrics for those services, i bet we would see a different story than what is shown on the "status" page.
In general, they seem bad at communicating relevant information. Just looking over at emails from Google, every single one of the last 10 emails they sent me was not relevant to me specifically.
> "[Important notice] Tax changes in Nepal" (yet I have never made a sale in Nepal)
> "Secure your Google Admin account with these best practices" (yet I already do all those things)
> "Preparing for the upcoming Google Ads change on October 31, 2023" (it's all about mediation waterfalls, and I'm 90% sure I don't use them, and 100% sure I don't know what they are)
https://cloud.google.com/service-health/docs/overview#how-pe...
When you suspect things are broken, check X (FKA Twitter).
Other devs will be talking about it before there's an official status page.
If we as an industry can't think of something better (cough honest status pages cough) … can we at least transition these tweets to Mastodon.
If your connector between your status monitor and the service breaks you'll have some subset of users panicking and causing problems (or asking for refunds for outages) when the service was up the entire time.
3rd party services are the only ones that you'll get a "more honest" but not always correct view of what the actual status is.
Where I work, just about any non-false-alarm incident ends up on the status page in a timely manner. There's nothing stopping the likes of AWS from doing the same except for culture.
Well, maybe not at AWS scales though, if they have thousands of everything =/
You can easily check it manually. Search for "EBS" and see a bunch of people talking about EBS timeouts or whatever. That was what I was getting at.
But yeah scraping is harder now.
> You can easily check it manually.
You cannot. Twitter's site is plagued by redirect loops. If you work around those, these days /search just redirs to the login page. You can view single tweets, but there won't be any replies. (I have no idea if the site formerly known as Twitter is still rate-limiting views, or if they canned that.)
It is unusable if you're not actively logged in, and some of us have no desire to give away a phone number just to see AWS's true status.
8:18 PM
APP UPDATE: Global: Google Kubernetes Engine Nodepool Upgrade Failures
Incident began at 2023-10-02 11:29 (all times are US/Pacific).Summary: Global: Google Kubernetes Engine Nodepool Upgrade Failures
Description: A mitigation has been rolling out and we are assessing its effectiveness. We will provide an update byTuesday 2023-10-10 12:00 US/Pacific with current details.
Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested but may...
I'm also incredibly annoyed at them displaying time in PDT. I genuinely don't understand why they decided on that instead of doing something normal like UTC or detecting my timezone. Especially annoying every six months because Europe and the US don't do Daylight Savings Time changes at the same time, so for a week or two there's an additional hour I have to account for.
To think, I once was so in love with it. This kind of crap, and imagining the live-site chaos makes me so glad to have noped-out when I did. With my modest lifestyle, I probably passed on life changing money but I also have 1/10th the stress and anxiety I used to.
Maybe it’s time to yolo with a regular container that just restarts on failures, ha…
The only negatives about Nomad is the Hashicorp license drama that has happened recently and persistent storage can be a pain in the ass.
They pulled the rug out on a lot of people and people tend to hold grudges. Community edition is peanuts compared to using an open source tool you _could have_ fixed your own bugs with by creating PRs or adding features with PRs. I used to be a huge hashicorp fanboy... _used to_.
Nomad is simple on the surface and could have been a great tool - but it is basically unusable on its own without tighter integration with Hashi’s own tools (eg Vault). Configuring and maintaining all those things is nontrivial and ends up being more annoying (for a lesser end result) than just using a fully managed kubernetes cluster like GKE
It’s just a bug in something most people don’t use.
HashiCorp Nomad is a perfectly easy and flexible orchestrator (not only containers, also VMs, random binaries, Firecracker, WASM, etc.) that is production grade. It has a number of advantages over Kubernetes (being drastically easier to deploy, use and maintain; having integrated native templating with logic; being much nimbler but also being able to scale much more, etc. etc.) and it's a very good choice in many scenarios. Of course there's also disadvantages, most notably ecosystem size, but often that doesn't matter. Kubernetes is great and extremely powerful, but also very complex and straight up overkill for many organisations.
There is an unsupported community edition
* It can't work down batch jobs in priority order [1] -- a feature even the ultra-old university cluster schedulers had
* Sometimes job starts can be in arbitrary order, in the API and the UI, making it difficult to find the most recent ones [2]
* Nomad seems to store pretty much everything in its Raft state, making it extremely slow to startup and operate (including GUI) if some rogue process created 100k allocations. Cleaning that up seems to require nuking the Raft state, thus also losing all info about all jobs.
I don't know how well Kubernetes does in these areas.
[1]: https://github.com/hashicorp/nomad/issues/12792 [2]: https://github.com/hashicorp/nomad/issues/17742
I guess, my take away at this second is to really be skeptical of hype that I don't have years of operational knowledge and experience with. Which, to be fair, regarding my current tech du jour (well, 9+ years), I have a list of complaints that could span a short novela.
Nomad's mental model is hard to grasp. Or let's say is not as elegant. It's is nowhere near k8s. And k8s at least from a usage standpoint can be explained in under ten minutes. Pod, Deployment, Service, Ingress and you're pretty much done.
It's a little surprising how many folks are unaware of the non-Kubernetes orchestration options. Going full Kubernetes is rarely what organizations need. Getting CI/CD -> containers on servers and being able to configure the resources containers need, having the placement handled, etc. gets most of these folks 90% of what they want with much much less complexity.
- AWS ECS: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...
- ECS Anywhere: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...
Being able to mount secrets and configmaps into the container file system (without having to modify the container image to provide an entrypoint) definitely seemed to be one major advantage of kubernetes over ECS a few years back.
See a CDK example:
https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_...
You can setup something to mount configuration files stored in say S3 -> into an EFS volume that you attach to all ECS tasks, for example.
The problem with ECS is that everything, including service discovery, is an integration with another AWS service. Making it even more difficult to ever migrate, but it does support much of what folks use Kubernetes for.
It's a wonderful, observable, manageable orchestration layer that's easily and safely customizable if you're wise enough to treat it as such.
* Instance refresh to deploy new images.
* Every so often update your AMI to patch.
* Create an RDS database and hook the instances up to that if you need such things.
It's really hard to get more boring but works than this. It can be set up in a few hundred lines of terraform. I promise the code for your entire AWS account will be less than just the k8s YAML.
We do this but create custom images that have all their requirements installed via ansible.
Unfortunately there's now a push to move to GKE instead of GCE, for no real reason.
If your workload is extremely variable, something like k8s with auto-scaling makes sense. But a lot of applications (maybe the majority?) have very predictable loads and you don't always need new services.
When you're at the size of large corps with hundreds/thousands of services, then k8s is amazing. If you're not even close to that number, it's very likely premature optimization.
However I would ask the question (same question I have been asking since the first appearance of k8s) what do you need so much that it justifies this complexity?
Implementing a simple autoscaling system or using the provided solutions like AWS ASG was and is results in more reliable infra.
If you want to use cutting edge there Lamda function that most cloud vendors offer. That is also less complex option.
Edit: But I see your point - if their stateful workloads are not "migration/K8S-incident" ready, it's hard to suddenly build that overnight.
From an outsider’s perspective, it looks like in a 2x2 matrix of developer simplicity/complexity and operational simplicity/complexity, the current patterns all seem to be heavily biased for developer simplicity/operational complexity.
1. Is this assumption correct?
2. Does optimizing for another quadrant: developer complexity / operational simplicity make sense?
My intuition is that complexity in code can be managed far better than complexity in operations. Developers have abstractions, reusable libraries, unit tests/integration tests, etc. There may also be weird efficiencies that arise from having developers deal with some of these problems right from the design stage.
It seems kubernetes takes a problem and pushes it to fully to operations.
Is there a solution that takes this problem and turns it into a developer problem?
However, writing those tests is incredibly hard. It doesn’t matter if you approach it from a dev or ops angle. The system under test doesn’t only have side effects, it is side effects. You also cannot mock most things (in my opinion…), as that is either also very hard to instrument or straight up removes the test usefulness altogether. Imagine mocking the AWS management API for your integration tests. Not possible.
So what Dev calls integration or e2e tests, ops calls the dev environment. Works, but differently to how devs would do it. I don’t see an alternative.
Next, as much as knowledge siloes are being heralded as evil, they exist. Undoing siloes altogether isn’t possible. You’ll end up reerecting them elsewhere. Devs have their skill sets, and ops isn’t part of that. The opposite is also true. The intersection can be substantial, but never enough to have dev to it all alone. I don’t think that’s a bad thing either.
It’s great if you actually need all of this and consider those practices essential complexity. If you don’t - it feels like someone is shoving accidental complexity down your throat.
Turning it into a developer problem will not hide all these things or make them more manageable. It means you will reinvent k8s yourself, poorly.
Edit: “we are assessing its effectiveness” lol, yeah that sounds more like they they are throwing something at the wall to see if it sticks.
Mitigation can vary: an additional firewall rule to stop certain traffic, a rollback to known good version, temporarily redirect traffic away from impacted data center. A mitigation is part of an incident response mainly focused on stopping the pain to whoever are impacted, it may not always be a long term solution.