Major Azure Outage
twitter.com
twitter.com
If regions fail independently and a failure in 1-2 regions brought down my system, that's my fault. But if region failures aren't independent and a global outage such as this is possible - well, that's pretty bad.
What shouldn't happen ever is that your entire cloud goes down because somebody pushed a bad config change to a service that serves literally your entire cloud.
Microsoft clearly hasn't architected everything to be region-independent. There are things that will always be somewhat global, but Azure seems pretty bad at this. This isn't the first time even in the past 12 months that they've had a global outage.
Seems like they have tons of global dependencies within their services which cause these cascading failures rather often... Seems like only a few months ago we were reading about a global outage that affected auth?
Regardless: Godspeed to the engineers working to fix this.
I'm not in the cloud provider game, but it seems like it would be important to audit and ensure that there are no critical cross-region dependencies. I assume GCP and AWS do this regularly?
It seems like there are some things that have to be somewhat global (IAM comes to mind), but minimization of that seems important.
IMO this is the most embarrassing non-security thing that can happen to you as a cloud provider.
1. The other providers have a better track record 2. The other providers don't go down globally multiple times a year
Not sure how many outages are there on AWS.
I've run most everything in us-west-2 Oregon the last 5+ years and I can't remember a similar sort of outage there in that time-frame.
A widespread world-wide outage like is happening now on Azure is a red flag imo.
Yuck. Please tell me you don't do that anymore (unless for specialized workloads where you don't care if an instance loses data due to a shutdown).
One that does come to mind is the S3 outage a few years ago, which was essentially a global outage.
Everyone has down time, it's just often coordinated to a lot of people when it does happen. It's still generally less than when you try to self host on a cloud provider, it's just not your mistake that did it.
I'm not sure what the SLA is on a single region, but going down 0-2 times per year is a reasonable expectation, depending on the length of each one. If you want more, you have to have regional failover.
Azure is burning error budget in every region today and you would need to failover to a different cloud provider or your own datacenter.
If I'm interpreting this correctly, there was no plan you could implement solely in Azure that could have helped you today.
Worldwide outage? That's really not the case. Over thousands of machines in 3 years I haven't seen a single failure that spanned more than one region. Or even a region going entirely offline. At most you'll see one service affected.
I can't access anything from here though.
EDIT (23:37 Oslo/Norway): connectivity is restored for us now