> there was no way to fail back power without an outage, then them going off and building their own equipment.
Anywhere I can read about that?
Anywhere I can read about that?
Pretty sensible IMHO - I live in a country with a reliable electricity grid, and outages due to UPS malfunction are about as common as power outages.
[1] https://www.datacenterdynamics.com/en/news/aws-develops-its-...
It doesn't take long without cooling to cook equipment to the point of failure or reduced reliability.
7 to 16kW per rack is common even in these older colo facilities.
And there never would have been enough UPS to make up for not enough replacement breakers on site.
I thought most DCs just pause the cooling until the generator comes up, rather than running the cooling on battery power?
They are expected to last that long, but if the batteries are on year 4 of their 5-year life, that may not happen. What also may not happen is the generator starting up.
Or the automatic transfer switch (ATS) not working properly: it should be on either input Feed A or Feed B, but when it tries to throw itself over (making a loud kah-chunk sound), it gets stuck in between—this happened to us once.
"The perversity of the universe always tends toward a maximum." — Finagle's Law of Dynamic Negatives
Fun story, small IT company had a natural gas generator installed at the new office they built out. Power went out for the first time, lights go out, generator kicks on, lights turn on, lights turn off. Long story short the electrician didn't have the right breaker size in the truck and used a small one. Building was empty when it was tested which is why bigger DCs have a load generator to test at load.
https://cloud.google.com/blog/products/compute/google-joins-....
In third party data centers you're co-located with other customers that have mostly comodity server hardware which won't have those options, they generally (at least in my experience) all provide UPS power as part of the facilitiy so you have it already and something like a fire if it did happen, would impact lots of other customers who likely don't have redunancy between data centers or fire compartments.
Additionally depending on your exact setup (it's possible you may have dark fibre directly out of the facilitiy, but often not) you're likely to be uplinked through powered equipment at the same facility, that wouldn't have the extra UPS power either.
Plus as others mentioned, the loss of cooling is the biggest problem. In fairness maybe that would buy you a few minutes for the kindof automatic switchover they talked about. But that's quite a bit of co-ordination to have your own power, to not rely on anyone else being powered and then to be able to co-ordinate your shutdown before the room gets too hot - and not be at risk of overheating the room for others.
Plus there are safety issues like emergency power out for fire fighters etc which again, I'm sure Google could deal with at a facility-wide level, but if you're a small fish in a much bigger data center its harder to co-ordinate that kindof thing.
So while I'm sure it's all possible, there are obvious headwinds in many directions.
IIRC, it's about 30 minutes—irrelevant of room size.
This is because while smaller rooms have less air volume as a 'buffer', they also you can't fit much gear. While huge data centres have huge volumes of air, they also have lots of gear.
So there's a coincidental proportional relationship what you (as a rule of thumb) have 30 minutes before things cook, regardless of room size. (At least for 'general purpose' computing: GPUs are probably another matter.)
The link someone else put about overriding faults might be it, if OP misremembered the problem.
And yet here we are, reading an article about "a total loss of power [...] following a reportedly simultaneous failure of four [...] switchboards serving all of Cloudflare’s cages. This meant both primary and redundant power paths were deactivated across the entire environment."
Hence why modern servers and their associated networking equipment have dual power supplies which could be connected to two seperate UPS systems. It would be very unlikely to have them both fail at once. In a less important home/small business scenario typically one supply is connected to the UPS and the other is connected to the wall via a surge protector.
Also, in my opinion, all UPS manufactures make horrible UPS equipment. There can be so many different types of failures or problems or glitches. You don’t want a UPS leaking or exploding. Also, there are laws in some states that only allow so many pounds of batteries in a building.