Google Cloud region currently down due to water intrusion
status.cloud.google.com
status.cloud.google.com
https://www.theregister.com/2023/04/26/google_cloud_outage/
There is mention of a fire as well.
Also, why batteries in a datacenter? When you implement a flush() command at the lowest level you're faced with two choices: 1) actually write to disk, then return from the call, 2) write to some cache/RAM and have just enough battery locally to ensure that you can write it to disk even if all power goes out.
Then there's the other problem of surviving long enough between a power interruption and diesel generators starting up. But this is a smaller problem, rebooting all instances in a datacenter is less bad than losing some data that was correctly flush()ed by software. Bad flush() behaviour can result in errors that cannot be recovered from without a complicated manual intervention (for example if it causes corrupted and unreadable database files).
Everything serious in the telecom/ISP infrastructure sector has a big -48VDC battery plant, or preferably separate A and B side -48VDC battery plants, to provide a significant buffer between power going Grid --> AC-to-DC Rectifiers --> Equipment, and when a generator can start up, warm up, and transfer switch does its job.
Even if a bunch of servers don't have any UPS or battery backup because they're designed to tolerate individual node (or whole rack, or whole row failures) the core network equipment in a datacenter will still have a huge battery plant.
Ideally if you have a chilled water loop for cooling you do not want it anywhere near your big-ass racks of batteries. Or near the racks that contain the rectifiers and DC breakers, distribution bus bars.
If you look at the battery racks in a traditional telco CO in the US for instance you will see that all of the cabling and batteries are a minimum of 1 foot off the floor, so that the whole place could theoretically flood and the DC distribution would remain unaffected. Same principle that applies to very traditional setups with wet-cell 2V lead acid batteries also applies to more modern things if building from scratch.
Think like a fiber termination demarc vs an in-cluster mesh.
By square footage maybe 95% of a google DC might have no UPS or battery backup but the core network for things like routers and DWDM transport equipment absolutely will have such.
If they were unlucky enough that the burst cooling loop met with the battery plant for the core gear in a building or small campus of buildings....
They create 3 separate arrays of batteries in each back. Each array represents a power phase, A-B-C.. if I remember correctly, each array has a number of low voltage/2000 amp batteries connected in series to make up for a 2000amp 480 volt leg on the other end.
In a tier 4 plus+1 datacenter, they have 4 battery rooms and 4 generators for each data pod. You have a primary generator and UPS battery set, and a backup generator set for each pod. And then that generator set has its own primary and secondary backup set. The end result is that they can work on any piece of equipment without interrupting power. In the event they lost the primary set or needed to take it offline for maintenance, they have the whole secondary redundant set to fallback on.
The servers on the received on the power cord after it passes the switchgear never know that there has been power source changes on the other end.
I think this is sort of big.
A few years ago I implemented a top to bottom ISO27k1 ISMS for a client handling extremely sensitive and mission-critical data for industry.
One risk I recommended controls for was that of a fire and/or flood at their primary datacentre for their client-facing offerings - this datacentre. I’ve experienced the misery of a datacentre oops myself, firsthand, twice, and it’s a genuine risk that has to be mitigated.
At my insistence, I had them burn hundreds of man-hours ensuring that they could failover to a new environment in a different datacentre with a bare minimum of fuss, as what I arrived to was an all the eggs in one basket situation. It took a fair bit of re-engineering of how deployments worked, how data was replicated, how the environment was configured - but they got there, and the ISMS was put into operation, and was audited cleanly by a reputable auditor, and everyone lived happily ever after.
Except… they were acquired by private equity. Who had no truck with all of this costly prancing about with consultants and systems. Risk register? Why do we need this? What value does it add today? ISO27k1? Don’t be silly. We have that certificate. You don’t need it. Dev team, ops team, leadership — almost everyone — ejected and replaced with a few support staff.
I see their sites are down.
I am sure that my name is currently being damned in a boardroom in Chicago, as as the person who warned of this, I am likely seen as responsible.
In that sense nothing ever changes. Anyway, kudos to you for staying the course on topics like these.
The fact is, most cloud providers offer multiple regions, which have the capability of giving you more geographic redundancy than most companies that operate in their own datacenters have.
Whether you choose to adopt a multi-region or multi-datacenter architecture is really orthogonal to whether you choose cloud or on-prem.
A nice thing about EC2 is that you're getting a pretty dumb, predictable service. There have been multi-zone or global control plane issues but the physical metal has bona fide redundancy between zones/regions.
They will arrange/move workloads such that any one customer will only see an outage in one 'zone'.
Clearly that didn't work here,
> Google designs zones to minimize the risk of correlated failures caused by physical infrastructure outages
And they have stated that the flood "caused a multi-cluster failure".
> Zones should be considered a single failure domain within a region.
(—GCP's documentation.)
Zones are an artificial construct.
By definition, AWS availability zones do not share fault domains, other than a geographic region up to hundreds of miles wide. Even for services used by multiple AZs, such as transit to the Internet and other regions, there are two transit centers operating in separate fault domains.
In contrast, many GCP Zones share the same physical datacenter. What they are actually providing you are simply different racks, rows of racks, or rooms in a single physical facility. Caveat emptor.
B4: https://storage.googleapis.com/pub-tools-public-publication-...
"GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating... "
Embarrassing.
That goes without saying at this point. More importantly, it’s proven worse than Azure.
On the other hand, Azure was(and still is) upfront about not having AZs - now that they have rolled out, hopefully those are not in the same building.
Alphabet is most ruled by arbitrary fiat of empire-builders.
Killing products and footgun changes are standard.
Someone in my freshman college dorm decided to use one as a clothes hanger hook and broke the thermometer in there. The sprinkler damaged the entire floor with water and the floor below had spotty rain as well.
The fire department came and was mainly concerned about evacuating everyone rather than shutting the water off.
The water is typically chemically treated and has been sitting there for years as well -- very nasty stuff.
The fire department is always going to prioritize safety of life, and after all it’s not their stuff getting soaked.
They won't hesitate to smash your stuff or break down your walls either.
Being in a fire is no joke. You've got to be crazy to think that your stuff is important. It's not.
They're boys with toys that they don't frequently get to use and they work for the government. Follow the incentives. They'll do their jobs but they don't give a lot of fucks about things like "unnecessary property damage" and "other people's financial well being" and anything else not written in their KPIs.
I used to drive tow truck. I can't count the number of cars they totaled peeling the roof off (granted some were totaled anyway) because that was easy and cutting a door off was hard. And don't get me started on them and their stupid stands they use to prop shit up in the most questionable of ways...
Good. I want you to save me and my family from dying in a fucking fire. This thread is obsessed with saving Funko Pop collections for some reason??
Most of their calls are mundane stuff. And they leave a pretty decently wide path of destruction in doing that. We're talking like mundane situations where there is no urgency and no need to tear shit up in the interest of time.
I once arrived to a minor rollover after the cops but before fire. Nobody injured. Occupant trapped because she was a large lady and couldn't release her seatbelt upside down and was having difficulty unlocking the car because side curtain airbags.
I offered to flip the car and treat it like a lockout. "Customer" was fine with it. Cop was iffy. FD showed up, didn't want to hear it, broke the window, unlocked the car, opened the door, cut her belt rather than release it and dropped her on her face and then had difficulty getting her out. Now I get that they have "procedures" but this seems like a forest for the trees situation.
Or they'll show up, shut down two lanes for a minor fire on the shoulder and not move the trucks until the car is loaded on a tow truck and gone. Supposedly it's to keep them safe from being hit by traffic. Meanwhile here I am not blocking traffic to recover shit that broke down.
Sure, they'll save a life if the situation presents itself but they sure don't care about being tidy about it.
You sound really butthurt.
https://www.youtube.com/watch?v=rO0ioCCiEe8
Firefighters arrive in 5 minutes.
Your fingerprints won’t survive the fire!
You will not care about your stuff when you're in jail for negligent manslaughter.
Hey Dang, thanks for cleaning up the thread. One thing to note is that the title is not correct. The entire region is not currently down, as the regional impact was mitigated as of 06:39 PDT, per the support dashboard (though I think it was earlier). The impact is currently zonal (europe-west9-a), so having zone in the title as opposed to region would reflect reality closer.
Finally, there's lots of good feedback on this thread and on the previous one (https://news.ycombinator.com/item?id=35711349), so we obviously have a lot of lessons to learn.
Was there a lot of anxiety? Panic? Or was it just a “woof that sucks. Time to follow a checklist and then do a bunch of paper work” ?
What I’m curious about is what it feels like on a team at a company like Google when there is a major system failure.
Personally, I wasn't part this time for the actual mitigation of the overall Paris DC recovery, as I was busy with an unfortunate[0] side effect of the outage. These generate more anxiety, as being woken up at 6am and being told that nobody understands exactly why the system is acting this way is not great. But then again, we're trained for this situation and there are always at least several ways of fixing the issue.
Finally, it's worth repeating that incident management is just a part of the SRE job and after several years I've understood that it is not the most important one. The best SREs I know are not great when it comes to a huge incident. But, they're work has avoided the other 99 outages that could have appeared on the front page of Hacker News.
Based on that thread it sounds like only AWS guarantees that their AZs are in physically separate DCs, while for Google and Microsoft AZs could be in separate buildings of the same DC facility.
It's incredibly rare for multiple AZs to go down at once, especially since they are more than a few miles apart from each other.
> Each Availability Zone is separated by a meaningful physical distance from other zones to avoid correlated failure scenarios due to environmental hazards like fires, floods, and tornadoes.
https://docs.aws.amazon.com/wellarchitected/latest/reliabili...
Very much so - Australia only got a second region this year, so if your work required data to remain in Australia, you just had to hope that ap-southeast-2 didn't have a major issues. I'm sure there are plenty of other countries with only a single region.
If I have to think too hard and do too much work about how I lay applications out, I might as well just rent in a colo.
[edit: at least in regions where Google HAS its own datacenters, e.g. "us-central-1? yes. europe-west9? maybe not"].
That does not imply that Search / Drive / Gmail runs on top of Google Cloud.
Instead aim for uptime in a few regions, and load balance your users to regions that are healthy.
That design is far cheaper for both google and for you - and, in the typical case, users still get nice low latency to a local datacenter, and only in the rare failure case might they have to wait for latency to some other region.
Other cloud providers mostly just vaguely put things in another part of the building and say it’s “a separate AZ” but as GCPs woes highlighted that’s corner cutting that bites badly when the whole building has a problem.
In many cases in AWS an availability zone is actually composed of multiple datacenters, each with their own redundancies. This may not be true for smaller regions, but in large ones it definitely is. In those cases, losing an entire datacenter would maybe take out a percentage of instances in that AZ. This has happened before and our production systems barely noticed other than provisioning new nodes to replace the failed health checks.
I think you misunderstand Google's infrastructure. I'm guessing that each GCP zone is actually a Borg Cell (see: https://storage.googleapis.com/pub-tools-public-publication-... ). Borg cells tend to be isolated from eachother in many ways in the physical layer (networking and management being a big one, not sure about power). So networking or machine management for an entire zone could go down and not affect other cells. Changes also tend to get pushed on a per-cell basis when they are Google wide rollouts.
I believe GCP recommends to replicate data cross regions (https://cloud.google.com/architecture/framework/reliability/...).
Also see: https://cloud.google.com/architecture/disaster-recovery#regi...
Obviously every cloud vendor recommends replicating data between multiple regions, but fact of the matter is that a lot of cloud services work much easier with redundancy within a single region than multi-region redundancy.
I think Google designs their datacenters for their own needs and expect you (a product running in their DCs) to distribute by region. Almost products at Google will be operating in multiple regions given the reach of most of our services, so DC design followed that need.
Based on GCP's docs, they still think region separate is better. Not sure why you wouldn't just do that?
If there is a catastrophic event (a large tornado hit AWS us-east-2), those buildings are pretty close to one another and both likely would be taken out, right? So you could lose multiple AZs since they are physically located so close to one another?
It says "reserved for future use" but other docs mentioned "physical zone separation": https://googleapis.dev/java/google-api-services-compute/alph...
Like, Target does not compete with Amazon. They have a totally different home delivery model that is not in the same category of reliability or service.
It's annoying.
No random online eshop can offer next day delivery across half the world unless they already have a logistics chain of 100,000 truck drivers spread across the world. But Amazon can.
Likewise, no cloud provider has enough data centers to offer multiple separate data centers in the same city, for hundreds of cities around the world. But Amazon does.
Any competitor can't offer amazons level of service until they get to amazon scale... Which they never will.
But for the service team I worked for, our AZ-evacuation story wasn't great at the time and it took us tens of minutes to manually move out of the AZ, but at least there wasn't a customer-visible availability impact. Once we did it was just monitoring and baby-sitting until we got the word to move back in, I think it was 1-2 days later.
If you operate on AWS you work with the assumption that an AZ is a failure domain, and can die at any time. Surprisingly many service teams at AWS still operate services that don't handle AZ failure that well (at the time). But if you operate services in the cloud you have to know what the failure domain is.
Ouch, hopefully none of the major services? I recently had to look into this for work (for disaster recovery preparation) and it seemed like ECS, Lambda, S3, DynamoDB and Aurora Serverless (and probably CloudWatch and IAM) all said they handled availability zone failures transparently enough.
I’m not that familiar with S3, but I never noticed any concerns with S3 during an AZ outage. I’m not at all familiar with Aurora Serverless or ECS.
For all AWS services you can always ask AWS Support pointed, specific questions about availability. They usually defer to the service team and they’ll give you their perspective.
Also keep in mind that AWS teams differentiate between the availability of their control and data planes. During an AZ outage you may struggle to create/delete resources until an AZ evacuation is completed internally, but already created resources should always meet the public SLA. That’s why especially for DR I recommend active-active or active-“pilot light”, have everything created in all AZs/regions and don’t need to create resources in your DR plan.
ECS Fargate's default is to distribute task instances across availability zones too but I'm assume if you use EC2 it might not be as straight-forward.
And that makes sense - I remember during the last outage that affected me it was a compute rather than data failure and the running stuff continued fine, just nothing new was getting created.
I have to assume it’s a fault that not even distributed services can paper over. Eg lots of crucial data in flight and they’re reluctant to drop it. Can an expert weigh in?
I love Google’s post-mortems. This one will be epic.
Right. But nobody forces GCP's customers to design their services to be tolerant of a single DC failure. In fact as a business, actively not designing for such tolerance is an attractive cost-cutting measure.
The logical units are regions and availability zones or the equivalent nomenclature in each cloud. One availability zone is expected to be one or more datacenters.
We have thousands of instances in AWS. I do not know - or care - where they are physically located(other than the region name, say, Oregon). I expect at most one availability zone to get impacted if a datacenter goes up in flames (and sometimes, just a portion of one). I mention in another comment that AWS has had issues before and production systems barely got impacted. And recovered with zero intervention - instances with failed health checks get replaced by brand new ones in whatever AZs are still operational.
> Data would be eventually consistent (once they find and plug the hard drives in)
At the level of abstractions cloud operates, no-one is plugging drives in – someone is, but you can never see it.
Most cloud workloads use network attached storage - when you can even see the logical drives (SaaS offerings may not even have that abstraction). We don't know (or care) how many physical hard drives exist, or where they are. Latency requirements probably dictate that they are close to the actual instances, but there's usually data replication going on even across DCs.
In addition to that, at least in AWS, if you have saved any volume snapshots at all, they will be in S3. This data will be replicated and underlying systems can even use it to restore lost or corrupted data without you even noticing and sometimes even without a recent snapshot, as storage keeps track of what blocks have been rewritten since the last snapshot. In a particularly bad case you might have to do a restore.
In almost a decade and number of volumes in the 6 digits (no clue how many drives that is!) we never had a single volume fail on AWS. Some got into a 'degraded' state and then recovered.
We haven't had any failures on GCP either. In the case of GCP, even faulty hypervisors are transparently worked around - we never notice other than some audit logs saying the VM was moved. They even preserve the network connections. AWS requires a stop/start to do the same, but your VM will be up and running in a different hypervisor (sometimes a different datacenter) in a couple of minutes, with all the storage.
Mind you, AWS promises eleven nines(!) of durability for S3.
When you do have locally attached storage, it's treated as ephemeral and it's gone if the instance restarts.
> I have to assume it’s a fault that not even distributed services can paper over.
If a single datacenter fails, since it _should_ be at most one AZ(this case seems to be different) that will depend on how the application is architected. Requests in flight will obviously fail, how big of a deal depends on the problem domain. For most web apps, this will cause a retry and that's the end of the story, others will be specifically engineered to deal with receiving multiple messages or dropping messages. For example, if you need at most once delivery guarantees, you need to take extra measures
Not all applications can survive an entire region going down. Some can, but that usually raises costs if you are continuously replicating data across regions. If you do that, then you should be able to steer traffic to the surviving regions. You can do that old-school by changing DNS records, or you could have fanciers solutions such as global anycast loadbalancers and have a single IP worldwide that still goes to the closest healthy region.
Literally no concern here for anyone's safety or sanity in dealing with this.
I think it's safe to assume that most people feel empathy for others struggling, whether or not they type it out regularly. Then again, some AI evangelists have had me questioning that assumption lately.
If you’ve only ever used the cloud, you’re not necessarily aware of everything that’s involved at data centers. If you’re not familiar with them, I don’t think you’d know how many things can (literally) blow up in your face. If someone sees flooding, they generally aren’t thinking that it’ll lead to fires.
Anyway, just want to think that everyone generally has good intentions and just don’t know what’s ACTUALLY happening in the DC, or how much work it will be for the folks working in the DC to restore services.
Hopefully all the failsafes kicked in and worked and nobody was injured.
OVH's caught fire
What's next, us-east-1 gets hit by Godzilla?
Source: my startup (stupidly) hosted our entire infra in us-east-1 at the time. Was a …tough day
When we were still designing the site/systems/infra we would go to BigRock Ranch a lot for meetings and vendor interviews.
Its the datacenter at bigrock which has one wall which backups to the lake on site, and the DC is in the sub-level parking garage, and its rear wall was leaking due to the pressure from the lake on the wall.
So some of mine woule be
Chinese air usb gaps and phishing
What stories can you expound upon?
Call California and Meyers to see if you qualify for PG&E benefits.
Joking asside I hope we will get a nice postmortem with juicy civil engineering details.
We called them bugs because you literally had to go in and get the dead bugs out of your electrical system.
Now we can call it fishing because some pirate has sailed onto your datalake and is looking for sunken hashes.
What do you think the hourly for Cloud Architect/Data lake power boy level 1 should start at?
Startlingly accurate in this case.
I think “unknown network” definitely accurately captures what hyperscalers are selling. :)
2) Not be able to use it
3) Company continues to pretend this doesn't happen on the regular
What do you want them to say? "Hey we have X breakdowns but please, pay!"
Or just cut down on nonsense in some way.
If they can figure out how to make nonsense, they can figure out how to dial it back.
Someone else managing your shit so you don't have to is a market in just about every industry, and it makes a ton of sense in tech where things don't even have to be on the same continent to work (or very specifically NEED to be on another one if you're international).
There's a ton of companies that have jumped to the cloud that probably shouldn't have, and even more who should've jumped, but not nearly as much as they did. Still it's a useful service.
Now of course it being a useful service, that also happens to be so obscenely expensive to start up barely anyone does it, means it comes with all the miserable obfuscation, bullshit fine print, total lack of support, and every other horrible thing we've come to expect from the modern world, but "It's just someone else's computer!" isn't changing any minds.
Either they already knew that or they never cared.
Cloud Computing I agree is a usual concept that marketing and experts often disagree about.
We're allowed a little humor, damnit.