Global Google Kubernetes Engine Outage
status.cloud.google.com
status.cloud.google.com
Status pages are rarely dynamic and updates require blessing from upstairs. And more often than not complete outages are referred to as "degraded performance affecting some users".
If there is a networking outage, everything on a status page should be red but then that looks bad for PR. So you just set “networking outage” but everything else is green even though everything is realistically down.
Some context before I go on is that reliability is often measured by mapping critical features to services and degradation. This gets more challenging as a feature starts to map to more than a couple services and those services begin to have dependencies. When your reliability on average can be measured in its number of nines opposed to its significant preceding digits your signal interpretation game has to step up significantly. These two situations make it infinitely more complex to state whether a given service degradation in a chain of services is truly having external customer impact at a given time. That's why a human needs to make the call to update the status page and why status page availability numbers are different from internal numbers.
I spend a good portion of nearly every sprint hunting down systemic issues that'll pop up across the ecosystem of services from a birds eye view. Often, knowing whether external customer impact will be felt for this series of errors relies heavily on knowing the current configuration of services in a chain, their graceful failure mechanisms, what failure manifests as client side, and whether that failure is critical to an SLA.
I have not, in my history of reliability engineering, seen anyone object to updating the status page for political reasons.
The status page is tied to public SLAs = impact on $$$. Internally you can track anything. What's public is the problem.
The status pages purpose is generally to head off a flood of customer reported issues. This is why you'll usually see issues that affect a broader subset of users on that page.
And how can I as a customer calculate this? We're not going to sue each time there's a breach of SLA to get the real data. Whatever the status page says will trigger customers to decide if they should claim SLA credits. A lower number (delayed update of the status page) will skip payouts or reduce it.
> The status pages purpose is generally to head off a flood of customer reported issues. This is why you'll usually see issues that affect a broader subset of users on that page.
That's what you assume and that's what it's supposed to be. It's long been abused otherwise. Amazon for example will require explicit approval to update the page. They and others have famously delayed updating the status page as late as they can get away with often attempting to not even call an outage. It will say something like "increased error rates".
Either your shit works, or it doesn’t. You do monitor, don’t you?
That then becomes a he said she said problem with the vendor you're claiming against. Does everyone have time for it? You will submit the SLA credit claim and chances are unless it's WAY off you'll accept the vendor's nerfed version and move on. Something is better than nothing.
I wasn't assuming what status pages are used for, I was speaking to my experience working in reliability engineering. I can't speak to Amazons practices as I've not worked there, but when I've seen this happen it's because we struggled to identify customer impact. The systems you're talking about are vaste and a single or even subset of applications reporting errors doesn't mean there's going to be customer impact. That's why I mentioned it usually takes a human that knows that system and it's upstreams to know if there'll be customer impact from a particular error.
I'd encourage you to read the wording of an SLA in a contract. They're often very specific in terms of time and the features they cover. Increased error rates tells me you'll probably run into retry scenarios, which depending on your contract may not actually affect an SLA. Error rates are generally an SLO or an SLI, which are not contractually actionable.
If I still had to deploy to Kubernetes, I’d consider this an outage.
1) actually using autoscaling in prod
2) using it for a stateful workload (with PV’s)
3) needing to scale that workload during this time window
4) using this specific k8s version
To be razor thin.
I’m honestly surprised they’ve reported the impact on this dashboard; but good to see they did.
Kubernetes 1.25 (in Danny Glover voice): I'm getting too old for this shit!
Need to deploy a dev copy of your cluster? How about the CI/CD pipelines that are blocked? Go sit on your hands and wait it out. Almost 2 days now.
Loads of pods don't even use PVCs. In the several hundred deployments I routinely manage, there's only a handful of PVCs, and they aren't exactly dynamically created and destroyed. I've got many GKE clusters and no workloads I'm running are affected, and I imagine most existing workloads in GKE clusters aren't affected. I'm then also doubly not affected, because I'm not running 1.25.
Would be interesting to hear from actual users how serious this is.
The announcement mentions specific version: "issue with Google Kubernetes Engine impacting customers using Kubernetes version 1.25"
Glad it's not me having to deal with this, Kubernetes is still a black box of magic in many respects and there's probably more than a few abstraction down to the actual disks.
> anything that already exists will be fine and continue operating.
Any pod that already exists will be fine, but that doesn't extend to services, as they may autoscale volumes along with pods. I'm not sure many users on such an antiquated kube version would have done that, but if no one did, there would probably not be a notice.
The same can be said of many smaller companies where uptime can be maintained cheaper and more consistently by offloading ops work to a cloud computing company.
What would that look like for your company compared to the cloud?
Can I steal this?
I think it's really apt.
"Hardware" meaning that it's very inflexible, "software" meaning that it is comparatively flexible, "wetware" meaning that it inherently lacks order. ;)