Microsoft shelves its underwater data center | Hacker News Reader

Microsoft shelves its underwater data center

tomshardware.com

32 pointsnamanyayg16 comments

> “I’m not building subsea data centers anywhere in the world.” She later added, “My team worked on it, and it worked. We learned a lot about operations below sea level and vibration and impacts on the server. So, we’ll apply those learning to other cases.”

It doesn't sound like they really discontinued the project. It sounds like it got moved to another team.

The article they got the quote from has some more quotes from the interview [1]. In context it sounds to me like they lost a lot of R&D budget ("have to get more focused") and as a consequence don't pursue subsea data centers any more. But they learned a lot that is applicable to other projects they are still running.

They mention liquid immersion, which is a less extreme take on the idea of trading lower failure rates for lower maintainability.

1 https://www.datacenterdynamics.com/en/news/microsoft-confirm...

The article alludes to the technique for dealing with failures inside a sealed underwater container: you never actually replace hardware but instead work around failures by reprovisioning services onto different nodes. Over time more and more nodes will fail until the container becomes exhausted and needs replacing.

That therefore begs the question: what are the narrowest points of failure in a system like this? For example it would be naive to have 140 fungible compute nodes if they all went through a single networking switch. Node workloads can be moved but a switch failure makes the entire container useless.

This works physically as well. Marine vessels have sealed bulkheads such that a hull breach in one end doesn’t cause flooding in the other end.

How do these vessels fan out all the other points of failure and what are the error rates in those components?

It's worth noting that redundant networking gear and redundant power are the norm even in normal data centers. Having an entire row of racks go offline because of a router failure is already considered unacceptable and well-mitigated.

Their biggest point of failure is probably the land connection. Looks like it's one big cable bundle. Those get damaged all the time from sea life, fishing, people throwing anchors where they shouldn't etc. Land-based datacenters have the same issue with excavators damaging power or network lines, but it's easy to run multiple lines far apart from each other when you are on land.

Wouldn't it be easier to just submerge some big heat exchanger?

At the end of the day, you're still wasting heat whereas you could heat a whole district with it.

But the point is that it is passive and works all year long, which I suspect wouldn't be the case of a residential heating system.

With an improvement of 87.5% to reliability compared to land-based data centers.

I wonder what factors led them to discontinue the project.

And I wonder if they meant 187.5% (this is better) or 87.5% (this is worse).

That it's significantly cheaper to deal with the failures than it is to prevent them.

I would expect price and practicability of physical intervention

Sure, they are less likely to fail in the stable environment, but you can't get to it until you pull up the entire vessel. On land 5.9% of servers required a technician to walk over there and replace components. At sea 0.7% of servers failed irrecoverably.

On the other hand 0.7% dead hardware sounds like a marginal cost. What likely killed it was the cost to deploy and recover them, plus all the hassle with sea cables for data and power. Building glorified warehouses with redundant power and AC is likely a lot cheaper than deploying these metal tanks in the ocean.

Less noise and dust down there - I think the air is purged with nitrogen as well so no oxidization. They could do the same thing cheaper on land and recycle the waste heat. Edit: just read the article - they also state temperature stability as a factor.

Having the servers more easy to access and repair also makes them more likely to fail. Also spaces that humans can enter require a bunch of regulation that takes up space and adds to the expense. So the math could easily work out in favor of a sealed box of computers that no-one is allowed to access and any broken servers just get switched off.

Some more discussion about the project in 2020 when they presented some of the findings:

https://news.ycombinator.com/item?id=24470954

Without knowing the security measures in place, it strikes me as a ripe vulnerability to any determined nefarious actor to destroy or infiltrate this submersible data centre.

such a data center would have been under a lot of pressure to perform

Reply on news.ycombinator.com