Liquid and immersion is the new cool at Supercomputing '22
theregister.com
theregister.com
But imho especially immersion cooling is a dead end. Contact problems with cabling lower reliability because the isolating coolant creeps between the metal surfaces. Maintenance is unbelievably convoluted, not only do you have to drain all the fluid out to pull a machine, you will need to clean each and every plug, socket and pin you touch because the contact surfaces will be coated with isolating coolant. This could only be fixed with a complete redesign of everything, so that all or at least most contacts are "dry". Using fluid-tight plugs is very expensive, and anyways, most ideas there won't work for important components such as CPU and RAM.
Liquid cooling will fare better, because you can strike a balance between high-power components like CPU and GPU where you slap a liquid cooler on. And low-power components like most of the board you just cool by air as usual. But even there, maintenance is somewhat of a hassle, in addition to all the cabling you also need to deal with piping, leaks, drippage (which aren't supposed to happen, but trust me, they do). I think the only field where this is really going to be used is HPC for applications where floorspace is limited. E.g. if your network cabling must be within a certain length limit of your central network node, so you need to cram all your racks for the supercomputer you are building within some 20m circle or the like.
Less top-of-the-line HPC and datacenters are better served by just limiting power density and continuing air cooling as usual. Datacentre floorspace is expensive, but not _that_ expensive, so you will always fare better (and usually more environment-friendly) with air cooling.
To my understanding, DCs use air conditioning systems for cooling. These are already heat pumps - why would it be more difficult to just pump that heat into district heating instead of outside air?
Economically the greenhouse would have a distinct advantage of using free heat, and already having enough electricity wired nearby.
This, of course, only makes sense during cold seasons, or somewhere like Alaska or Sweden. During hot months, 40-50°C of heat seem completely useless.
Maybe next to a sea such waste heat could help evaporatively desalinate seawater, while cooling back to reusable temperatures.
For those regulations, there is also the important aspect of free cooling, i.e just using outside air as a medium or using an air-air heat exchanger like a Tokyo Wheel. Mandated heat reuse (without appropriate exceptions) would make those illegal, however in certain situations those are the best (environmentally and financially) methods of cooling. E.g. in a moderately hot summer you can use 20-30°C outside air without a problem, but if you would mandate heat reuse at that time, you would have lots and lots of excess heat on your hands with nobody buying it. So while some regulation will be coming, I hope and think that it wouldn't really issue a strict mandate to reuse heat and allow free air cooling, which would exclude direct water cooling.
>so you will always fare better (and usually more environment-friendly) with air cooling.
Sorry but this is not true. One of the key drivers of liquid cooling IS efficiency. Liquids are for more effective at heat transfer than air. Additionally, a liquid can carry a lot more heat. Yes, you can run higher density, you can also indirectly free cool servers year-round, even in tropical climates.
Unless your radiator is outside of the building itself, you're dumping that waste heat right back into the system over time.
There's a lot of things you can do outside of a server chassis that you cannot do inside of a chassis. You can have heat exchangers with far more surface area than what you are volume constrained to inside of a chassis. You can run evaporative cooling towers. If you're in the right climate, you can even radiate that heat to space.
>You're actually doing not much more with liquid cooling than just changing where the heat concentration is located.
You're using a more efficient heat transfer medium which does a lot of things for you. You can downsize your chiller tonnage, reduce/eliminate CRAC/CRAH units, and all the UPS required to keep those systems running. All of this translates to CAPEX and OPEX savings.
>Unless your radiator is outside of the building itself, you're dumping that waste heat right back into the system over time.
No one is suggesting that you can plonk some servers into a tank of liquid and the thermal energy magically disappears. Of course, there has to be a thermodynamic "sink" that you are dumping to... the atmosphere, the ocean, the ground, space, etc. Liquid cooling is more efficient means of transferring heat from source to sink.
You use a liquid to transport the heat from denser compute (lowering costs), then run it through a high efficiency radiator (eg, drip/spray towers like with industrial AC). Your compute is denser and your interface with the air is more efficient.
Liquid cooling is, without question, more /effective/. It makes things cooler, it has a far higher capacity for heat transport.
However, with /efficiency/, you do weigh the results against the effort put in: A solution is more efficient if the quotient result/effort is higher. However, in case of cooling, there are two ways to measure effort: energy expended or money expended. In the case of energy efficiency, yes, liquid cooling is better, because of the aforementioned higher effectiveness of heat transport, lesser energy expenses for pumps/fans and easier energy reuse. However, financially, all that liquid handling, custom piping, pumping, coolant, etc. is by far more expensive than a few air ducts and fans. Even with air-air heat pumps, air cooling is still less expensive, so better financial efficiency. That's why everybody is doing it as a default atm.
Google championed this over a decade ago. They figured that the cost of staff, spare parts, diagnostics, etc... outweighed the cost of the hardware. If you factor in the complexity of liquid cooling, it might not make sense to repair hardware even a single compute node costs north of $20K.
First, we are designing smaller transistor leading to leakage and thus increasing chip thermal design power (TDP). We are also using bigger chips and chiplets also increasing TDP.
TDP on a GPU went from 200w to 1 kW in the past 5 years and even CPU are now close to 500w. At 25k a pop avoiding throttling due to heat restriction is a must and is becoming incredibly hard with air cooling as air is an amazing insulator.
Second, networking is extremely expensive with the cost of top-end InfiniBand cable close to 1000$ a foot. This means you want to cram your CPU /GPU as close as possible to keep the connection electrical if possible and minimize optical cable lengths. This also decreases latency and increase cluster performance. On a 50 000 cable deployment, the saving can be … significant.
Third, we are now seeing the emergence of foundational model that requires government lab levels of interconnected compute. This is a new paradigm in the private sector and consume power in the 20-30 MW range. An increase in efficiency of 10% (from 1.2 to 1.1 PUE) brings huge saving on the power cost side plus important environmental benefits. It also becomes much easier to recuperate heat.
That’s why we’re seeing major announcement at OCP i.e. meta going the liquid cooling way.
I personally believe that’s where things are going and we are building a facility to optimize for the above points: https://www.qscale.com/
Wait... wat? I thought InfiniBand just used fibre!
The 30m pricing is almost the same: https://www.naddod.com/products/18231.html
1m $1,087
10m $1,124
30m $1,196
It was a terrible idea back then and still seems bad today. I'll be interested to see what the next cycle brings, if it's back to spot cooling or if immersion really is the way of the future.
Liquid cooling is nice on paper, but as others have noted, it is messy AF.
Another huge issue that you don't normally think aobut is staffing... some data centers are in the middle of nowhere. It is hard enough to just fine skilled enough labor to work reliably... getting them trained up to do maintenance on wet systems is a whole another level of pain.
It is great for small systems... not so good at scale. It is amazing how proper airflow with fans can solve most cooling needs. Even if the outside temps are high as well.
[1] https://twitter.com/SenTedCruz/status/1564441153306406912
Don’t get me wrong, the Novec line and it’s ancestors and siblings like the stuff that cooled the old Cray supercomputers… are the sort of Chemical wizardry that had chemical engineering on the prospective career list when I was in high school. I just remember seriously investigating using them for a project regardless of the price and coming across some sort of chemical safety red flag. So if anyone knows/recognises what I’m half remembering, feel free to share the details!
Pre-covid we paid roughly $225 for 20 liters, not bulk pricing. Not quite apples to apples as 3M Novec is a 2 phase coolant, we went with single phase.
Fortunately I had the pleasure of using these coolants in high school when I was hired to help build a commercial scale ASIC farm in the midwest - we used about 200 gallons of 3m Novec 7000. Which is insane since that coolant cost around $180/gal even at a steep discount ($36,000 in total). And that's after the shipment arrived with 10% of the bottles "empty" since the container sat in a hot port - coolant simply slipped past gaskets on the glass bottles.