The OVHcloud fire still smolders (2024) | Hacker News Reader

The OVHcloud fire still smolders (2024)

datacenterdynamics.com

20 pointsvarun_ch15 comments

I don't quite get the mean and accusatory tone of this article.

It's obvious to everyone that stuff went seriously wrong and that they lost a lot of money when their hardware caught fire. And OVH was way more transparent about what happened and why than what you usually hear from Amazon/Google/Microsoft about their outages. They quickly provided internal documents, video footage, and their own theory as to why a power supply caught fire. (And their info matched up with later-released firefighter heat camera footage.) Most likely, their own legal battle with their supplier of the USP that first received maintenance and then a day later caused the fire is still ongoing, so they probably aren't allowed to say much anymore, or else it would be considered interference with their own lawsuit.

They refunded customers for the downtime and, additionally, also granted extra refunds to customers who did not have proper backups. That's more generous than my experience with AWS & GCloud, who just categorically refused any data-loss-related refunds even though we followed their best practices with a multi-AZ RDS deployment and automated snapshots to S3. When I saw data loss with a cloud storage provider that claimed to be more durable than an atomic bunker, they just shrugged it off and since it wasn't viable to sue them for it, they just got away with not paying. OVH tried the same with a customer where automated backups turned out badly, and then lost in court and were ordered to pay up. In my experience, OVH is behaving like everyone else, they don't stand out as especially bad here.

The article then gets accusatory with "Neither OVHcloud nor Klaba will talk about the SBG2 fire now, but they are both using the experience in surprising ways, having seen that national champions get special treatment." and "being a European tech champion makes you fireproof." but I just don't see that. They paid for the damages caused, changed their datacenter design to prevent these issues in the future, and then rebuilt the whole thing on their own costs. How is that "special treatment"? What could they have done better?

And then the article complains about datacenter operators not sharing enough details publicly. But if this is the treatment you get for being even mildly transparent, I can totally understand why everyone now keeps their lips sealed.

I don't think giving refunds to consumers who lost important data is really fair.

It's like my insurance going like "oops sorry to hear about that fire, here's your premiums refunded, bye now"

The original article says that they provided refunds in addition to the regular fees. So then it would be akin to your insurance refunding the premiums AND paying you some extra, which is kinda what I would expect an insurance to do.

That said, we are talking about consumers here who rented a bare-metal or VPS server, where back-ups are typically not included. AWS will also not refund you anything for data on the locally-attached SSD with EC2.

No, it’s not the same. When you go for hosting and don’t do backups, the hosting company doesn’t offer you an insurance against data loss. Now, you could get one yourself. However, good luck with your premiums when the insurance company finds out that you didn’t have backups. What’s the bottom line here? You’re responsible for your backups.

Others already commented about the accusatory tone of the article, but I want to focus on:

>But public sector agencies were selecting it as a way to support Europe’s native tech sector, and also to take a step towards digital sovereignty

I care about neither, but I did -and still do - use OVH. They were never good, but they were always cheap. They provide good VPS and dedicated bare metal servers. They were not a "cloud" like DigitalOcean or AWS, with things happening magically in the background.

Therefore, there are no backups included, by default servers come with RAID but I dismantle mine for more disk space (and do offsite backups).

Also, they are bigger than the article suggests. Even if they're not well known outside of Europe:

>As of 2016 OVH owned the world's largest data center in surface area.[3] As of 2019, it was the largest hosting provider in Europe,[4][5] and the third largest in the world based on physical servers.[6] According to W3Techs, OVH has 3.4% of website data center market share in 2024

Yeah scaleway is much more like a cloud. Most of their services are AWS compatible even.

Spoiler: The fire is in fact not still smoldering, the title is metaphorical

If you pay a company to do backups, is it unreasonable to expect those backups to be at least basic 3-2-1 backups?

I'd say you would hope to receive that, but in reality you don't. For Google Cloud and AWS, even a "multi-AZ" deployment might mean that it's just 2 servers in the same building, i.e. exactly what OVH is accused of here. For example, see this discussion about Google's Paris disaster:

https://news.ycombinator.com/item?id=35711349

and especially: https://news.ycombinator.com/item?id=35713001

And here for AWS multi-AZ losing data: https://news.ycombinator.com/item?id=4115937

I fully agree with you that OVH did a bad job. But so do all the other cloud providers. Cloud just means "someone else's computer" and you still need a watertight contract and the legal ability to sue them to make sure they uphold their end of the contract.

> And here for AWS multi-AZ losing data: https://news.ycombinator.com/item?id=4115937

That link is from 2012, 13 years ago.

AFAIK as of today AWS provides strong guarantees about the AZs being physically, network, and power isolated of each other.

They certainly say that. But it might turn out to be a lie, just like it did with OVH and Google.

I think that there's enough evidence that AZs are kms apart.

It is, but did the customers who lose data pay OVH for backups? I use OVH for at least 10 years now, and I never had any backups by them.

Despite the rebranding in 2019, they started as and still mostly are hosting providers, not the usual cloud.

OVH was not paid for backups. They sell access to VPS without backups

I stopped reading at "Never before or since has a complete data center burnt down." What other wild, easily refutable claims await? Offsite backups are a scam?

Reply on news.ycombinator.com