Hacker News Reader

Comment by worldofmatthew | Hacker News Reader

https://twitter.com/olesovhcom/status/1369504527544705025

"Update 5:20pm. Everybody is safe. Fire destroyed SBG2. A part of SBG1 is destroyed. Firefighters are protecting SBG3. no impact SBG4."

View on HN

> Fire destroyed SBG2

This is crazy.

SBG2 was HUGE and if this isn't a translation error on the part of Octave (which I could understand given the stress and ESL) I have a hard time fathoming what kind of fire could destroy a whole facility with nearly 1000 racks of equipment spread out across separated halls.

I'm really hoping "destroyed" here means "we lost all power and network core and there's smoke/fire/physical damage to SOME of that"

I can't even fathom a worst-case scenario of a transformer explosion (which does occur and I've seen the aftermath of) having this big of an impact. Datacenters are built to contain and mitigate these kinds of issues. Fire breaks, dry-pipe sprinkler systems and fire-extinguishing gas systems are all designed to prevent a fire from becoming large-scale.

Really glad nobody was hurt. OVH is gonna have a bad time cleaning all this up.

Just guessing, but maybe a fire suppression system going off could wipe out all the machines?

The couple datacenters I've been inside were small, old and used halon gas which wasn't supposed to destroy the machines. No idea how it works in big places these days.

A few years back there was an incident in Sweden where noise coming from the gas based fire suppression system going off destroyed hard drives [1].

1. https://www.theregister.com/2018/04/26/decibels_destroy_disk...

I've also seen a weird video (lost to time unfortunately), where someone showed that they could yell at their servers in a data centre and introduce errors (or something similar). Was very strange to see, but they had a console up clearly showing that it was having an impact.

https://www.youtube.com/watch?v=tDacjrSCeq4&fmt=18

Because of course it was Bryan :D

Brian just giving his opinion on anything would give hard drives errors... That guy loves to run his mouth (and I love to listen).

Actually, although the video is on Bryan Cantrill's YouTube account, that was Brendan Gregg.

I was merely the videographer! For anyone who is curious, I discussed the backstory of that video with Ben Sigelman.[0]

[0] https://www.youtube.com/watch?v=_IYzD_NR0W4#t=28m46s

I actually planned to include a link to this video in my original response but then left it out. Thanks for posting!

ING in Romania had the same issue: https://www.datacenterdynamics.com/en/news/noise-from-fire-d...

We had the same issue at a customer site. To add since the decibels were outside the rated environment the warranty was void on the harddisks and they had to be replaced even if they were not destroyed.

If a fire supression kicks in or the fire department shows up with their hoses, would they still say the fire destroyed it or just say destroyed due to fire and water damage?

Also, fire suppression system do fail. There was an infamous incident in LA for one of the studios. They built a warehouse to be a tape vault with tapes going back to the 80s. A fire started, but the suppression system failed because there was not enough pressure in the system. Total loss. Got to keep your safety equipment properly tested!

It wasn’t just “tapes going back to the 80s.” Those were just the media Universal initially admitted to losing. No, that building was the mother lode. It had film archives going back over 50 years, and worst of all — unreleased audio tape masters for thousands upon thousands of recording artists. The amount of “remastered” album potential that fire destroyed is probably in the billions of dollars, let alone the historical loss of all those recordings by historical persons that will never be heard due to a failure of a fire prevention system. Fascinating case study in why you should never put all your eggs in one basket.

https://en.m.wikipedia.org/wiki/2008_Universal_Studios_fire

In 1996 Surinam lost many of their government archives after a fire burned the building down.

I can find surprisingly few English-language resources for this (only Dutch ones); guess it's a combination of a small country + before the internet really took off.

This is why I'm such a fan of digitizing. If you have 1 film master you effectively have none. Do some 8K or 16K scans of that master and effectively manage the resulting data and you're effectively 100% immune from future loss in perpetuity.

Losing parts of history like that is tragic.

There's a problem with testing sprinklers: engaging them can be damaging to contents and even structures. So, we're talking about completely emptying the facility, then taking it offline to dry for a time. I've never heard about this being done to anything that was already operational (but I wasn't researching this either).

Isn't something like Halon used in data centers for that reason? That can probably be tested without damaging infrastructure.

Halon is pretty much banned for years now, other agents have been introduced. Sadly, making an actual full test of a gaseous extinguishing system (such as FM200 or Novec 1230) can be prohibitively expensive (mainly costs of the "reloading" of the system with new gas). Those are just mostly tested for correct pressure in tanks and if the detection electronics are working fine, making an actual dump would be very impractical (evacuation of personnel, ventilating the room afterwards etc.)

Good point. But, apart from cost of the gas, mentioned by sibling comment, that also isn't disruption-free. Gas-based systems are by definition (they displace oxygen) dangerous to humans. But yeah, not being able to run maintenance is orders of magnitude less problematic.

There are methodds of testing the water pressure in the pipes without actually engaging the sprinkler heads. It is part of the normal checks done during the maintenence/inspection a business is supposed to have done. In fact, one place I was in had sensors, and would sound the actual fire alarm if the pressure fell below tolerance at any time. The lack of pressure in the Universal vault was 100% unexcusable.

It’s common for sprinklers in parking garages in cold climates to be “dry” where there’s a bubble of air that needs to run through first before water shoots out from a non-freezeable source.

Want to test the system? Just turn off that water valve, hook up your air compressor and pressure test to your heart’s content.

No, "destroyed" looks like this: https://mobile.twitter.com/abonin_DNA/status/136953802824345...

EDIT a picture from earlier this night https://pbs.twimg.com/media/EwGqV17XMAMF_wa?format=jpg&name=...

I’d say destroyed is an adequate word, yes.

I'm really shocked, that's an incredible amount of damage. Lots of people lost data.

The pictures seem pretty clear that a lot of it is gone, judging by the blackened holes in the walls with firehoses being sprayed into.

This was effectively posted on the outages list as well by someone trustworthy. The pictures also look pretty bad from the outside.

Which outages list? Sounds interesting.

I presume they mean this: https://puck.nether.net/mailman/listinfo/outages

Ah, thanks!

OVH design their own datacenters, so it's possible that they missed something or some system or another didn't work as intended, thus the heavy damage.

They did not have a fire suppression system, only smoke detection. So yeah, they missed something.

Where did you get this?

https://www.ovh.com/world/us/about-us/datacenters.xml

They say different here: "The rooms are also equipped with state of the art fire detection and extinction systems."

https://www.ovh.com/world/solutions/centres-de-donnees.xml

and here

"Every data center room is fitted with a fire detection and extinction system, as well as fire doors. OVHcloud complies with the APSAD R4 rule for the installation of mobile and portable fire extinguishers and has the N4 certificate of conformity for all our data centers." https://us.ovhcloud.com/about/company/security

That's for its North America DCs, not European ones.

Which is understandable, as Halon based fire suppression systems have been illegal for quite some time :/

There are non-Halon systems, using agents such as FM-200. Those are not as toxic and do not destroy the ozone layer.

There are photos of the fire suppression system in one of their data centres in this (French) forum thread: https://lafibre.info/ovh-datacenter/ovh-et-la-protection-inc.... They have sprinklers, with the reasoning being that the burning racks' data is gone anyway if there's a fire, and at least sprinklers don't accidentally kill the technicians.

Sprinklers are only present in OVH Canada datacenter ! There are no sprinklers in Europe ones.

Given that they're headquartered in Europe and most popular there, why is the satellite location better? Is it because the Canadian data center is newer, because Canada has stronger regulations in this area, or something else? Also, does anyone know how the OVH US data centers compare?

According to a 2013 forum post by the now CEO of Scaleway, a competitor of OVH, it's due to North American building regulations that basically force you into sprinklers and stuff for insurance reasons.

Source in French: https://lafibre.info/ovh-datacenter/ovh-et-la-protection-inc...

> They have sprinklers, with the reasoning being that the burning racks' data is gone anyway if there's a fire

I think the real problem, per that post, is this:

>> They are simple sprinklers that spray with water. It has nothing to do with very high pressure misting systems, where water evaporates instantly, and which save servers. Here, it's watering, and all the servers are dead. It's designed like that. Astonishing, isn't it?

>> Obviously, they rely above all on humans to extinguish a possible fire, unlike all conventional data centers.

(all thanks to Google Translate)

This strikes me as a terrible safety system because even if a human managed to detect the fire, they have to make a big call: is the risk of flooding the facility and destroying a ton of gear worth putting out a fire? By the time the human decides "yes, it is", it may well be too late for the sprinklers.

> and at least sprinklers don't accidentally kill the technicians.

Not a real risk with modern 1:1 argon:nitrogen systems – the goal is to pump in inert gases and reduce oxygen content to around 13%, a point where the fire is suppressed and people can survive. You wouldn't want to be in a room breathing 13% oxygen for a long time, but it won't kill you.

All in all, it looks like this was a "normal accident"[1] for a hosting company that aggressively competes on price. The data center was therefore built with less expensive safeties and carried a higher risk of catastrophic failure.

[1]: https://en.wikipedia.org/wiki/Normal_Accidents

> They did not have a fire suppression system

I find it very hard to believe that that would pass code anywhere in the US/EU or most of the world. They may not have had sprinklers but that doesn't mean there isn't fire suppression.

According to a forum post discussing the building of said DC, with photos and such, there's no visible fire suppression system:

https://lafibre.info/ovh-datacenter/ovh-et-la-protection-inc...

The floors are plywood and it's built with wood frame construction.

I retract all my prior surprise that this place burned down.

Data centers frequently burn down, or are destroyed by natural disaster.

These days, fire suppression systems need to be non-lethal, so inert gasses are out. Water is too, for obvious reasons. Last I checked, they flooded the inside of the DC with airborne powder that coats everything (especially the inside of anything with a running fan). Once that deploys, the machines in the data center are a write off even if the fire was minor.

Most datacenters I’ve worked with in the last 5 years use water.

I wonder if they had batteries inside the containers. That can make a bad situation really worse.

For some context:

“SBG1, the first Strasbourg data center, consisting of twelve containers, came online in 2012. The 12 containers had a capacity of 12,000 servers.

SBG2 is a non-container data center in 2016 using its “Tower” design with a capacity of 30,000 servers.

SBG3 tower was built in 2017 with a capacity of 30,000 servers.

SBG4 was built in 2013 as several containers to augment capacity, but was decommissioned in 2018 and moved to SB3”

https://baxtel.com/data-center/ovh-strasbourg-campus

and a 2017 (!) article where they were planning to remove 1 and 2 because of power issues!

> OVH to Disassemble Container Data Centers after Epic Outage in Europe

> “This is probably the worst-case scenario that could have happened to us.”

> OVH [...] is planning to shut down and disassemble two of the three data centers on its campus in Strasbourg, France, following a power outage that brought down the entire campus Friday, causing prolonged disruption to customer applications that lasted throughout the day and well into the evening.

https://www.datacenterknowledge.com/uptime/ovh-disassemble-c...

It sounds like the plan was to shutdown 1 and 4, the latter of which happened and the former which did not.

It’s confusing as they say they’re shutting down 2 of the 3, but then there’s 4...

An easy mistake (in the quoted tweet) to make given the circumstances, but it probably should be 5:20am rather than pm.

Reply on news.ycombinator.com