Liquid-Cooled Supercomputers, to Trim the Power Bill | Hacker News Reader

Liquid-Cooled Supercomputers, to Trim the Power Bill

nytimes.com

5 pointsdigital559 comments

In my experience, RAM needs to be replaced VERY OFTEN in HPCs. I seem to remember reading somewhere that motherboards that go into a mineral oil bath are basically impossible to clean afterwards. This point seems important, since mineral oil is used here for its electrically insulating properties. There are many contact pads in the marrying of RAM DIMM to slot. Ergo, replacing sticks of RAM in such a supercomputer would probably be a nightmare. And this nightmare would likely be frequent in its occurrence.

I also take issue with the first sentence of the article. Submerging a home computer in mineral oil is a common practice. Their "technology" editor needs some more technology chops, it would seem.

Another note: Let's hope they aren't secretly using polychlorinated biphenyls (PCBs) in place of mineral oil to give themselves a boost. (see Wikipedia)

I've read a similar article sometime last year, about Intel testing immersion-cooled blades. That solution would make more sense IMO - blades are already designed to be a single FRU, so if any component were to die, you'd just pull the blade from its chassis. I imagine the process of safely disconnecting the fluid hookups could be integrated into the blade chassis packaging system.

Not just "one" model of Cray-at least the Cray-1, XMP, C90, T916/932, T3E…were cooled by liquid HFCs.

At five million+ $$ per system, the cost of using Fluorinert as opposed to mineral oil in the coolant cycle isn't even a rounding error.

If you spill mineral oil while swapping internal components, it's a trip & fire hazard. HFCs evaporate quickly, and aren't flammable.

You might be surprised at flourinert cost, when that was contemporary rather than ancient history I was looking into FC-40 and the like for some ham radio gear cooling (hey, why not?). Believe it or not, FC-40 retails today for about a kilobuck per 2-liter bottle. No I'm not kidding. So a couple thousand liter tank is about a megabuck. So its not a rounding error.

You could have picked a Cray T94 full of it when it was for sale on Ebay.

https://news.ycombinator.com/item?id=4560173

Coming from a EE background, sounds like total bogosity relying on the customers not knowing anything about electronics. Magically dripping oil on a piece of electronics makes it use half the power, yeah get back to me on that.

I never knew that adding massive capacitance to high speed signalling lines made their drivers draw less current (super sarcastic here, its the other way around)

I've dunked my share of power resistors, precision resistors, and the occasional semiconductor in oils of various sorts and nothing too exciting happens. Some electrolytic caps are not very well sealed and leak oil in and (conductive) electrolyte out. Some component marking inks are oil soluble and some inks are semi-conductive, which can be a problem.

It's not that the components themselves draw half the power, it's that you don't need to provide as much power to cool them. There's a typical rule of thumb in datacenter planning that for every 1W of compute power, you need 1W to cool it. If you can cut the cooling power, you can cut almost 50% of your datacenter cost.

Obviously you won't eliminate cooling cost entirely, and it's quite possible the overhead of replacing oil and extended maintenance times could offset these power savings.

Why not go for warm water cooling then (like SuperMUC[1] for example)?

Should get you the same benefits as submerging it in oil without most of the drawbacks.

[1] http://www.lrz.de/services/compute/supermuc/systemdescriptio...

Wouldn't you save a lot more money improving the COP of the refrigeration system from 1 or so, to something a little better? Even the cheapest poorly installed window air conditioner can do 3 or 4, and central residential air can do much better.

Reply on news.ycombinator.com