Google's Liquid Cooling | Hacker News Reader

Google's Liquid Cooling

chipsandcheese.com

I once saw an interview with the SVP who oversees Azure datacenter buildout or something like that and a thing that stuck with me was that he said his job got a lot easier when he realized he was no longer in the computer business, he was now in the industrial cooling business.

Reading this article immediately made me think back to that.

curious to see that interview, do you remember link?

It’s very odd when mainframes (S/3x0, Cray, yadda yadda) have been extensively water-cooled for over 50 years, and super-dense HPC data centers have used liquid cooling for at least 20, to hear Google-scale data center design compared to PC hobbyist rigs. Selective amnesia + laughably off-target point of comparison.

Hyper-scale data centers normally need not be concerned with power density, and their designers might avoid density because of the problems it causes. Arguably modern HPC clusters that are still concerned about density are probably misguided. But when it comes to ML workloads, putting everything physically close together starts to bring benefits in terms of interconnectivity.

LOLWUT? Hyperscalers and HPC data centers have been very concerned about power and thermal density for decades IME. If you're just running web sites or VM farms, sure keep racks cooler and more power efficient. But for those that deeply care about performance, distance drives latency. That drives a huge demand to "pack things closely" and that drives thermal density up, up, up. "Hot enough to roast a pig" was a fintech data center meme of 20 years go, back at 20kW+ racks. Today you're not really cooking until you get north of 30kW.

Name a hyper-scale data center where the size and shape suggests that power density ever entered the conversation.

My impression is a lot of modern data centres are total-power-constrained, but not really space-constrained.

At least, if you look at them on streetview a lot of them seem to be in the middle of nowhere, surrounded by miles of undeveloped scrubland. If Google's Henderson, NV data centre [1] needed more space they could simply buy out the adjacent car wrecking yard or pet crematorium, or reconsider the gigantic gatehouse and vast expanses of beige gravel.

Even in Belgium, with its higher population density [2] the car park is bigger than the data centre.

The layout makes me think they were told they could only have a certain amount of power, but essentially as much land as they needed. So they're concerned about power and thermals, but maybe not about power and thermal density so much.

[1] https://www.google.com/maps/place/Google+Data+Center+-+Hende... [2] https://www.google.com/maps/place/Google+Data+Center/@55.557...

From the article:

> Liquid cooling is a familiar concept to PC enthusiasts, and has a long history in enterprise compute as well.

And the trend in data centers was to move towards more passive cooling at the individual servers and hotter operating temperatures for a while. This is interesting because it reverses that trend a lot, and possibly because of the per-row cooling.

This abundance of marketing (not necessarily this blog post) is happening because of all the environmental chatter about AI and data centers recently.

Google wants you to know it recycles its water. It's free points.

Edit: to clarify, normal social media is being flooded with stories about AI energy and water usage. Google isn't greenwashing, they're simply showing how things work and getting good press for something they already do.

Google uses plenty of water in cooling towers and chillers. Sure, water cooling loops to the server may reduce the amount a little bit compared to fans, but this isn't "recycling" in any normal sense.

It's mostly a play for density.

The environmental impact of water usage seems way overblown to me.

Last year, U.S. data centers consumed 17 billion gallons of water. Which sounds like a lot, but the US as a whole uses 300 billion gallons of water every day. Water is not a scarce resource in much of the country.

Personally, I agree. That said, I think it might be worth considering the impact of water usage in local areas that are relatively water constrained.

To be clear all talk of water usage has been focusing on local usage which isn't well represented by national numbers.

Yeah. There are metrics for how water stressed a community is. We know where data centers have been built, and we know how these water metrics have changed over time.

The correct metric is something like, what's the probability that the launch of data center in a location results in nearby communities to drop significantly in these water metrics.

It seems like a very minor problem next to the huge super obvious one, electricity use.

Google almost certainly uses evaporative cooling towers, it’s almost twice as efficient as air cooled chillers by themselves.

We've basically been watching Google gradually re-discover all the tricks of supercomputing (and other high performance areas) over the past 10+ years. For a long time, websearch and ads were the two main drivers of Google's datacenter architecture, along with services like storage and jobs like mapreduce. I would describe the approach as "horizontal scaling with statistical multiplexing for load balancing".

Those style of jobs worked well but as Google has realized it has more high performance computing with unique workload characteristics that are mission-critical (https://cloud.google.com/blog/topics/systems/the-fifth-epoch...) their infrastructure has had to undergo a lot of evolution to adapt to that.

Google PR has always been full of "look we discovered something important and new and everybody should do it", often for things that were effectively solved using that approach a long time ago. MapReduce is a great example of that- Google certainly didn't invent the concepts of Map or Reduce, or even the idea of using those for doing high throughput computing (and the shuffle phase of MapReduce is more "interesting" from a high performance computing perspective than mapping or reducing anyway).

As somebody that worked on Google data centers after coming from a high performance computing world I can categorically say that Google is not “re-learning” old technology. In the early days (when I was there) they focused heavily on moving from thinking of computers to thinking of compute units. This is where containers and self contained data centers came from. This was actually a joke inside of Google because it failed but was copied by all the other vendors for years after Google had given up on it. They then moved to stop thinking about cooling as something that happens within a server case to something that happens to a whole facility. This was the first major leap forward where they moved from cooling the facility and pushing conditioned air in to cooling the air immediately behind the server.

Liquid cooling at Google scale is different than mainframes as well. Mainframes needed to move heat from the core out to the edges of the server where traditional data center cooling would transfer it away to be conditioned. Google liquid cooling is moving the heat completely outside of the building while it’s still liquid. That’s never been done before as far as I am aware. Not at this scale at least.

It's possible it never made it into production; but when I was helping to commission a 4 rack "supercomputer" circa 2010 we used APC's in-row cooling (which did glycol exchange to the outside but still maintains the hot/cold aisle) and I distinctly remember reading a whitepaper about racks with built in water cooling and the problems with pressure loss, dripless connectors, and corrosion. I no longer recall if the direct cooling loop exited the building or just cycled in the rack to an adjacent secondary heat exchanger. (And I don't remember if it was an APC whitepaper or some other integrator.)

There's also all the fun experiments with dunking the whole server into oil, but I'll give you that again I've only seen setups described with secondary cooling loops - probably because of corrosion and wanting to avoid contaminants.

The parent poster is just either extremely confidently wrong or talking about a very different project from the one in the linked article - here's an article from 2005 with Figure 1 dating from (according to the article) 1965 (!!) showing the same CDU architecture shown in the chipsandcheese article: https://www.electronics-cooling.com/2005/08/liquid-cooling-i...

I do think Google must be doing something right, as their quoted PUE numbers are very strong, but nothing about what's in the linked chipsandcheese article seems groundbreaking at all architecturally, just strong micro-optimization. The article talks a lot about good plate/thermal interface design, good water flow management, use of active flow control valves, and a ton of iteration at scale to find the optimal CDU-to-hardware ratio, but at the end of the day it's the same exact thing in the diagram from 1965.

yea I totally missed the CDU. I thought this was a project I had talked with a hardware person about a few years ago where there was no intermediate transfer and when I read the article I completely missed the section between the images. Rack level water cooling is interesting and I am sure they are doing some really cool bits on it but it’s not as revolutionary as a zero transfer system that I had thought they were talking through. I updated the comment to call out my error and reduce my excitement. =/

[I am still annoyed at how many people are dismissive of Google’s datacenter work simply because “severs have been water cooled before” which completely misses the point of datacenter level cooling. I also learned that AWS is doing this already, along with some elements of OVH] =)

> cooling is moving the heat completely outside of the building while it’s still liquid.

We have been doing this for decades, it's how refrigerants work.

The part that is new is not having an air-interface in the middle of the cycle.

Water isn't the only material being looked at, mostly because high pressure PtC (Push to Connect) fittings, and monitoring/sensor hardware has evolved. If a coolant is more expensive but leaks don't destroy equipment, and can be quickly isolated then it becomes a cost/accounting question.

> The part that is new is not having an air-interface in the middle of the cycle.

I wasn’t clear when I was writing but this was the point I was trying to make. Heat from the chip is transferred in the same medium all the way from the chip to the exterior chiller without intermediate transfers to a new medium.

The claim is that Google has larger pipes that go all the way out of the building. While mainframes have short pipes that go only to a heat exchanger on the end of the hack.

IMO, it's not a big difference. There are probably many details more noteworthy than this. And yeah, mainframes are that way because the vendor only creates them up to the hack-level, while Google has the "vendor" design the entire datacenter. Supercomputers have had single-vendor datacenters for decades too, and have been using large pipes for a while too.

Glycol is cheap and safe, but it has lower specific heat capacity and higher viscosity. So that's why water is still being used.

The next step is probably evaporative cooling, with liquid coolant ("freon") pumped to individual racks.

Not sure what Google specifically is using here, but PG25 (25% propylene glycol, 75% water) is somewhat common in data center applications. The glycol takes care of avoiding microbial growth.

"From the core to the edges of the server"—what does that even mean?

Unless Google has discovered a way to directly transfer heat to the aethereal plane, nothing they’re doing is new. Mainframes were moving chip and module heat entirely outside the building decades ago. Immersion cooling? Chip, module, board, rack, line, and facility-level work? Rear-door and hybrid strategies? Integrated thermal management sensors and controls? Done. Done. Done. Done. Richard Chu, Roger Schmidt, and company were executing all these strategies at scale long before Google even existed.

It has been done. See how the leonardo supercomputer is cooled. Closed circuit water that is passively cooled completely outside the building: https://www.youtube.com/watch?v=mckX_R0GeRY

As to MapReduce, I think you're fundamentally mistaken. You can talk about map and reduce in the lambda calculus sense of the term, but in terms of high performance distributed calculations, MapReduce was definitely invented at Google (by Jeff Dean and Sanjay Ghemawat in 2004).

Not quite. Google brilliantly rebranded the work of John McCarthy, C.A.R. Hoare, Guy Steele, _et al_ from 1960 ff. e.g. https://dl.acm.org/doi/10.1145/367177.367199

Dean, Ghemawat, and Google at large deserve credit not for inventing map and reduce—those were already canonical in programming languages and parallel algorithm theory—but for reframing them in the early 2000s against the reality of extraordinarily large, scale-out distributed networks.

Earlier takes on these primitives had been about generalizing symbolic computation or squeezing algorithms into environments of extreme resource scarcity. The 2004 MapReduce paper was also about scarcity—but scarcity redefined, at the scale of global workloads and thousands of commodity machines. That reframing was the true innovation.

CERN was doing the equivalent of MapReduce before Google existed.

Got a link? HEP data is really about triggers and widely distributed screening that's application specific AFAIK.

My main reference is the head of computing at CERN, who explained this to me. He gave some early examples of ROOT (https://en.wikipedia.org/wiki/ROOT) using parallel processing of the ROOT equivalent of SSTables.

SPMD was definitely a thing before MapReduce.

> MapReduce is a great example of that- Google certainly didn't invent the concepts of Map or Reduce, or even the idea of using those for doing high throughput computing (and the shuffle phase of MapReduce is more "interesting" from a high performance computing perspective than mapping or reducing anyway).

The “Map” in MapReduce does not originally stand for the map operation, it comes from the concept of “a map” (or, I guess, a multimap). MapReduce descends from “the ripper”, an older system that mostly did per-element processing, but wasn't very robust or flexible. I believe the map operation was called “Filter()” at the time, and reduce also was called something else. Eventually things were cleaned up and renamed into Map() and Reduce() (and much more complexity was added, such as combiners), in a sort of backnaming.

It may be tangential, but it's not like the MapReduce authors started with “aha, we can use functional programming here”; it's more like the concept fell out. The fundamental contribution of MapReduce is not to invent lambda calculus, but to show that with enough violence (and you should know there was a lot of violence in there!), you can actually make a robust distributed system that appears simple to the users.

I believe Map in MapReduce stood for "map" function, not a multimap- I've never heard or read otherwise (maps can operate over lists of items, not just map types). That's consistent both with the original mapreduce paper: """Our abstraction is inspired by the map and reduce primitives present in Lisp and many other functional languages. We realized that most of our computations involved applying a map operation to each logical “record” in our input in order to compute a set of intermediate key/value pairs, and then applying a reduce operation to all the values that shared the same key, in order to combine the derived data appropriately"""

and with the internal usage of the program (I only started in 2008, but spoke to Jeff extensively about the history of MR as part of Google's early infra) where the map function can be fed with recordio (list containers) or sstable (map containers).

As for the ripper, if you have any links to that (rather than internal google lore), I'd love to hear about it. Jeff described the early infrastructure as being very brittle.

> and with the internal usage of the program (I only started in 2008, but spoke to Jeff extensively about the history of MR as part of Google's early infra) where the map function can be fed with recordio (list containers) or sstable (map containers).

I worked on the MapReduce team for a while (coincidentally, around 2008), together with Marián Dvorský, who wrote up a great little history of this. I don't think it was ever made public, though.

> As for the ripper, if you have any links to that (rather than internal google lore), I'd love to hear about it. Jeff described the early infrastructure as being very brittle.

I believe it's all internal, unfortunately.

> We've basically been watching Google gradually re-discover all the tricks of supercomputing (and other high performance areas)

Google isn't claiming to have invented water cooling. This article recaps their talk at Hot Chips where they showed some details of their implementation.

Data center cooling is also a different beast than supercomputer cooling. Datacenter cooling operates at a larger scale and has different needs for maintenance operations like maintenance cycles.

There are also some interesting notes in there about new things Google is doing, like the direct-die cooling.

Water cooling is a big field. Data center operations is a big field. There is interesting content in the article.

I think that's a little unfair.

Building out an idea for a bespoke supercomputer and making an iteration of that idea that applies to globally scaled workloads is a different thing. They are building computing factories, as is Amazon, MS, Facebook, etc.

That said, PR is PR, and companies always crow about something for their own reasons.

It's not so surprising when considering googles history coming from inexpensive commodity hardware. It's pretty similar to how it took decades for x86 servers and operating systems to gain mainframe functionality like virtualisation.

https://blog.codinghorror.com/building-a-computer-the-google...

Once again IT comes full circle.

[bri3d pointed out that I missed an element of this. There is a transfer between rack level and machine level coolant which makes this far less novel than I had initially understood. See their direct comment to this]

I posed this further down in a reply-to-a-reply but I should call it out a little closer to the top: The innovation here is not “we are using water for cooling”. The innovation here is that they are direct cooling the servers with chillers that are outside of the facility. Most mainframes will use water cooling to get the heat from the core out to the edges where traditional where it can be picked up by the typical heatsink/cooling fans. Even home PCs do this by moving the heat to a reservoir that can be more effectively cooled.

What Google is doing is using the huge chillers that would normally be cooling the air in the facility to cool water which is directly pumped into every server. The return water is then cooled in the chiller tower. This eliminates ANY air based transfer besides the chiller tower. This is one being done a server or a rack.. its being done on the whole data center all at once.

I am super curious how they handle things like chiller maintenance or pump failures. I am sure they have redundancy but the system for that has to be super impressive because it can’t be offline long before you experience hardware failure!

[Edit: It was pointed out in another comment that AWS is doing this as well and honestly their pictures make it way clearer what is happening: https://www.aboutamazon.com/news/aws/aws-liquid-cooling-data...]

So every time they plug in a server they also plug in water lines?

Looks like it. New server means power, internet, and water.

just like humans.

[I am not a current Google Employee so my understanding of this is based on externally written articles and “leap of faith” guestimation]

Yes. A supply and return line along with power. Though if I had to guess how its setup this would be done with some super slick “it just works” kind of mount that lets them just slide the case in and lock it in place. When I was there almost all hardware replacement was made downright trivial so it could just be more or less slide in place and walk away.

Maybe similar to a gasoline hose breakaway

https://www.opwglobal.com/products/us/retail-fueling-product...

Interestingly, entire supercomputers have been decommissioned [1] due to faulty quick disconnects causing water spray.

So you can get a single, blind-mating connector combining power, data and water - but you might not want to :)

[1] https://gsaauctions.gov/auctions/preview/282996

https://amphenol-industrial.com/products/liquid-cooling-syst...

CPC is in this market too https://www.cpcworldwide.com/Liquid-Cooling/Products/Blind-M...

Non-spill fluid quick disconnects are established tech in industries like medical, chemical processing, beverage dispensing, and hydraulic power, so there are plenty of design concepts to draw on.

You can see the male quick disconnect fittings for the liquid cooling at each corner of the server in this photo:

https://substackcdn.com/image/fetch/$s_!8aMm!,f_auto,q_auto:...

Looks like the power connector is in the centre. I'm not sure if backplane connectors are covered up by orange plugs?

it's not the center one, it's the side ones. Center seems to be a power supply.

I remember reading somewhere that they don't operate at the level of servers; if one dies they leave it in place until they're ready to replace the whole rack. Don't know if that's true now, though.

It does sound like connections do involve water lines though. As they are isolating different water circuits, in theory they could have a dry connection between heat exchanger plates, or one made through thermal paste. It doesn't sound like they're doing that though.

Definitely not true for these workloads. TPUs are interconnected, one dying makes the whole cluster significantly less useful.

It has not been true for a LONG time. That was part of Google early “compute unit” strategy that involved things like sealed containers and such. Turns out that’s not super efficient or useful because you leave large swaths of hardware idle.

In my day we had software that would “drain” a machine and release it to hardware ops to swap the hardware on. This could be a drive, memory, CPU or a motherboard. If it was even slightly complicated they would ship it to Mountain View for diagnostic and repair. But every machine was expected to be cycled to get it working as fast as possible.

We did a disk upgrade on a whole datacenter that involved switching from 1TB to 2TB disks or something like that (I am dating myself) and total downtime was so important they hired temporary workers to work nights to get the swap done as quickly as possible. If I remember correctly that was part of the “holy cow gmail is out of space!” chaos though, so added urgency.

    > part of the “holy cow gmail is out of space!” chaos

This sounds like an interesting story. Can you share more details.

I'd love to work in a datacenter at that scale sometime, it sounds like it's like working in a warehouse where you get a list of orders, servers to remove and pick up, but at the scales of the Googles et al, that's hundreds of server replacements a day, and production lines of new servers being built and existing ones being repaired or decommissioned.

It's a fascinating industry, but only in my head as the only info you get about it is carefully polished articles and the occasional anecdote on HN, which is also carefully polished due to NDAs.

Maybe we can declutter things if they get PWoE(power and water over ethernet) or just a USB-W standard.

It worked for MONIAC.

Great, now I’ll have to figure out if my USB Hi-Flow cable is slowing down my USB Full-Flow drink bottle refilling.

It's not. Full flow is less than hi-flow... Your bottle might slow down other fills on the bus though :p

Related, I vaguely recall a concept from a Tesla charger for semis that proposed a charging cable with active coolant flowing through it as well to keep the wire cooled.

edit: https://www.teslarati.com/tesla-liquid-cooled-supercharger-c...

This is why I risk jokes on HN. Normally they're frowned upon but it's worth it to find interesting tech that meets the eccentric idea.

And a 12V battery.

This was before I was born, so I'm hardly an expert, but I've heard of feeding IBM mainframes chilled water. A quick check of wikipedia found some mention of the idea: https://en.wikipedia.org/wiki/IBM_3090

When our mainframe in 1978 sprung a leak in its water cooling jacket it took down the main east/west node on IBMs internal network at the time. :-). But that was definitely a different chilling mechanism than the types Google uses.

Having to pre chill water (via a refrigeration cycle) is radically less efficient than being able to collect and then disperse heat. It generates considerably more heat ahead of time, to deliver the chilled water. This mode of gathering the heat and sending it out, dealing with the heat after it is produced rather than in advance, should be much more energy efficient.

I don't know what surprises me about it so much, but having these rack-sized CDU heat-exchangers was quite a surprise, quite novel to me. Having a relatively small closed loop versus one big loop that has to go outside seems like a very big tradeoff, with a somewhat material and space intensive demand (a rack with 6x CDUs), but the fine grained control does seem obviously sweet to have. I wish there were a little more justification for the use of heat exchangers!

The way water is distributed within the server is also pretty amazing, with each server having it's own "bus bar" of water, and each chip having it's own active electro-mechanical valve to control it's specific water flow. The TPUv3 design where cooling happens serially, each chip in sequence getting hotter and hotter water seems common-ish, where-as with TPUv4 there's a fully parallel and controllable design.

Also the switch from lidded chips to bare chips, with a cold plate that comes down to just above, channeling water is one of those very detailed fine-grained optimizations that is just so sweet.

Much of the Google use of liquid chillers was protected behind NDAs as part of its "hidden advantage" with respect to the rest of the world. It was the secret behind really low PUE numbers.

Do we know if other hyperscalers also use liquid chillers to achieve very low PUE values? I think I saw photos from xAI's new data center and there was liquid cooling.

I don’t know of any Co-lo types that did it, Microsoft had some creative ideas about things like that but I don’t know if they brought any to fruition. I suspect there are still opportunities there.

I don't think this comment is accurate based on the article, although you cite personal experience elsewhere so maybe your project wasn't the one that's documented here?

> What Google is doing is using the huge chillers that would normally be cooling the air in the facility to cool water which is directly pumped into every server.

From the article:

> CDUs exchange heat between coolant liquid and the facility-level water supply.

Also, I know from attaching them at some point that plenty of mainframes used this exact same approach (water to water exchange with facility water), not water to air to water like you describe in this comment and others, so I think you may have just not had experience there? https://www.electronics-cooling.com/2005/08/liquid-cooling-i... contains a diagram in Figure 1 of this exact CDU architecture, which it claims was in use in mainframes dating back to 1965 (!).

I also don't think "This eliminates ANY air based transfer besides the chiller tower." is strictly true; looking at the photo of the sled in the article, there are fans. The TPUs are cooled by the liquid loop but the ancillaries are still air cooled. This is typical for water cooling systems in my experience; while I wouldn't be surprised to be wrong (it sure would be more efficient, I'd think!), I've never seen a water cooling system which successfully works without forced air, because there are just too many ancillary components of varying shapes to successfully design a PCB-waterblock combination which does not also demand forced air cooling.

> > CDUs exchange heat between coolant liquid and the facility-level water supply.

Oh interesting I missed that when I went through in the first pass. (I think I space bared to pass the image and managed to skip the entire paragraph in between the two images so that’s on me.

I was running off an informal discussion I had with a hardware ops person several years ago where he mentioned a push to unify cooling and eliminate thermal transfer points since they were one of the major elements of inefficiency in modern cooling solutions. By missing that as I browsed through it I think I leaned too heavily on my assumptions without realizing it!

Also, not all chips can be liquid cooled so there will always be an element of air cooling so the fans and stuff are still there for the “everything else” cases and I doubt anybody will really eliminate that effectively. The comment you quoted was mostly directed towards the idea that Cray-1 had liquid cooling, it did, but it transferred to air outside of the server which was an extremely common model for most older mainframe setups. It was rare for the heat to be kept liquid along the whole path.

The CDUs are essentially just passive water to water heat exchangers with some fancy electronics attached. You want to run a different chemical mix outside to the chillers as you do on the internal loop, it also helps regulate flow/pressure and leak detection with auto cutoff is all fairly essential.

Running direct on facility water would made day to day operations and maintenance a total pain.

    > not all chips can be
    > liquid cooled.

Why not? It's just a heatsink except with water running through cavities within it, instead of a fan sitting on top of the heatsink.

One of the biggest problems with water cooling, especially on boards that weren’t designed for it, can be passive components which don’t usually have a heatsink and therefore don’t offer a good surface for a water block, but end up in a thermal design which requires airflow - resistors and FETs are common culprits here. Commodity assemblies are also a big problem, with SFPs being a huge pain point in designs I’ve seen.

The problem is often exacerbated on PCBs designed for air cooling where the clearance between water cooled and air cooled components is not high enough to fit a water block. Usually the solution when design allows is to segment these components into a separate air cooled portion of the design, which is what Google look to have done on these TPU sleds (the last ~third of the assembly looks like it’s actively air cooled by the usual array of rackmount fans).

Indeed, and the problem is once you've committed to fans and liquid cooling you can reduce the complexity and plate size massively by just cooling the big wins (CPU/GPU). I've actually seen setups where they only cold plate the GPU and leave the CPU and it's entire motherboard on air cooling.

Messy.

I wonder if you could just put a conventional heatsink in there to cool the air inside the box?

You would have a liquid block on the CPU but you'd also have a heat sink on top that transfers heat from the air to the coolant block, working in reverse compared to normal air cooling heatsinks. The temperature difference would cause passive air circulation and the liquid cooling would now cool both the CPU and the air in the box, without fans.

Seems like something someone would have thought about and tested already though.

Not really practical it wouldn't transfer much energy at all. Let's say that your coolant comes in at 30 degrees c, well if your air is 40° and you've got no fans you can do the maths but it may as well be 0.

I was imagining the coolant comes in at a lower temp like 20 and maybe keeps the air from going above 40.

It doesn't have to do that much, but maybe you're right. I'm sure they'd be doing this if it was practical, being able to onit thousands of fans would probably save a pretty penny both on hardware and electricity.

If you're blasting enough air around to cool a 600W GPU, you don't care if your GPU's power connector dissipates 10W under certain circumstances - the massive airflow will take care of it.

Take that airflow away and you have to be a good deal more careful with your connector selection, quality control and usability or you'll risk melted connectors.

Water-cooling connectors and cables isn't common, outside of things like 250kW EV chargers.

It's exactly those kind of problems (though these systems will be SXM like).

It's interesting because I've never seen mainframes do water to water (though I'm sure that was possible).

The only ones I've ever seen do water to compressor (then gas to the outdoor condenser, obviously)

Most ES/9000 series and earlier stuff had water to water SKUs. I remember seeing an installation with one that exchanged heat into a facility loop which went through another water to water exchanger to a pond-fountain chiller system.

Starting with S/390 G4 they did a weird thing where the internals were cooled by refrigeration but the standard SKUs actually had the condenser in the bottom of the cabinet and they required raised floor cooling.

They brought water to air back with the later zSeries, but the standard SKUs mimicked the S/390 strategy with raised floor. I guess you could buy a z196 or a ec12 with a water to water cabinet but I too have never seen one.

> they are direct cooling the servers with chillers that are outside of the facility

That is exactly what the Cray Y-MP EL that I worked with in the 90s/2000s did.

> super-dense HPC data centers have used liquid cooling for at least 20

Hasn't this just been for things like rack doors and such?

In the last ~two generations of servers it seems like now there's finally DLC (direct liquid cooling) into the actual servers themselves (similar to the article). Intel kind of forced that one on us, with their high-end SKUs. This has been a pain becuase it doesn't fit into legacy datacenters as easily as the door/rack-based systems.

I won't say which server vendor it is, but I've put in more than one service ticket for leaking coolant bags (they hang them on the racks).

In theory data center cooling is simple - CPUs run at 60-70C, while the outside ambient is usually 30C or less, so heat should just "flow downhill" with a bit of help from fans, pumps, etc.

The problem with using air cooling to get it there is that the humans who run the data center have to enter and breathe the same air that's used to cool the computers, and if your working fluid gets too hot it's quite unhealthy for them. (we run our hot aisles at 100F, which is a bit toasty, and every third rack is a heat exchanger running off the chilled water lines from the outside evaporative cooler, modulo a heat exchanger to keep the bird shit out)

We're not going to be able to pump much heat into the outside world unless our working fluid is a decent amount hotter than ambient, so when it gets reasonably warm outside we need to put chillers (water-to-water AC units) in the loop, which consume energy to basically concentrate that heat into a higher-temperature exhaust. When it's really hot outside they consume quite a bit of energy.

If the entire data center was liquid cooled we could have coolant coming from the racks at a much higher temperature, and we'd be able to dump the heat outside on the hottest days without needing any chillers in the loop. As it is we have some liquid cooled racks running off the same chilled water lines as the in-aisle heat exchangers, but the coolant temp is limited by the temperature of our hot aisles, which are quite hot enough already, thank you.

15 years ago IBM installed a supercomputer at ETH Zurich that used 60C hot water as its coolant, with a heat exchanger to the building’s hot water system (which is typically somewhat less than 60C) https://en.m.wikipedia.org/wiki/Aquasar

I wonder if we will eventually have people entering datacenters in heatsuits to eke out some extra efficiency gains from just needing to cool people.

> CPUs run at 60-70C, while the outside ambient is usually 30C or less, so heat should just "flow downhill" with a bit of help from fans, pumps, etc.

This isn't how you think of heat flow. The CPUs run at a given power. Their temperature will depend on the ambient temperature, and on the thermal impedance between them and the ambient. If the thermal impedance is too high, they'll be too hot and die.

> CPUs run at 60-70C

When properly cooled. If cooling is insufficient they can easily get much hotter.

> TPU chips are hooked up in series in the loop, which naturally means some chips will get hotter liquid that has already passed other chips in the loop. Cooling capacity is budgeted based on the requirements of the last chip in each loop.

Of course, it's worth noting that if you've got four chips, each putting out 250W of power, and a pump pushing 1 litres of water per minute through them, water at the outlet must be 14°C hotter than water at the inlet, because of the specific heat capacity of water. That's true whether the water flows through the chips in series, or in parallel.

Hm... but in the case when the chips are in serial, the heat transfer from the last chip will be less than when the chips are in parallel, because the rate of heating is proportional to the difference in temperature, and the water starts at a lower temperature for the parallel case for this last theoretical chip.

One way to characterize the cost of cooling is entropy production. As you say, cooling is proportional to difference in temperature. However, entropy production is also proportional to temperature difference. It's not my field, but it looks like an interesting challenge to optimize competing objectives.

Yes, but water is constantly moving in a loop. It's not like you use water to cool chip #1, and then it moves to chip #2, it's constantly moving, so temperature delta isn't that much.

in their first serial design, that's exactly what it was doing.

They mean it doesn't sit around and wait until it's hot, which then makes it "used" and unable to cool something else. It keeps moving through the first chip and is only a small percent saturated.

depends how fasts it is moving - but it's all moving at the same speed. The last chip will get less cooling than the first chip, proportional to the amount of cooling the previous chips received. Delta T doesn't lie.

Whether it's significant or not, I can't know - but you want it to be significant, otherwise it's less efficient.

> depends how fasts it is moving - but it's all moving at the same speed. The last chip will get less cooling than the first chip, proportional to the amount of cooling the previous chips received. Delta T doesn't lie.

The watts of cooling per chip will be the same. The last chip in the loop will be a little warmer, but not by much in a reasonable setup. The difference in temperature between each chip and the water running across it will be the same.

And if you take a weak water supply and then split it to run in parallel, you can end up with a significant heat gradient across each waterblock which doesn't sound great either. If you have 4 high power chips please don't limit them to .25 liters per minute each.

> Whether it's significant or not, I can't know - but you want it to be significant, otherwise it's less efficient.

Keeping your fluid cool is good for long term reliability. And if you're doing that, then every block is getting cool fluid and the other details about loop layout won't matter.

It moves fast enough that the temperature delta between first and last chip isn't significant. Not too fast, though, because if the pump is running full-tilt it becomes a major heat-source itself.

> but you want it to be significant, otherwise it's less efficient.

No? You want a large delta between heat-source and water, as well as water and whatever is on your cool side. Essentially just between hot and cold sides.

Delta between heat-sources is irrelevant unless you start cherry-picking some very odd combinations.

In steady state the power put into the chip is removed by the water (neglecting heat transfer away from the cooling system). The increased water temperature on entering into the cooling block is offset by a correspondingly higher chip temperature.

It just means in series that some of your chips get overcooled in order to achieve the required cooling on the hottest chip. You need to run more water for the same effect.

I can imagine a setup where multiple series of slower cooler water converging into a faster warmer stream, and the water will extract an equal amount of heat away from all the chips whether upstream or downstream.

While there is some truth to your comment, it has no practical engineering relevance. Since energy transfer rate is proportional to temp difference, therefore you compute the flow rate required, which is going to be different if the chips are in series or in parallel.

If you're heating up the water by 10 or so degrees on typical computer hardware, I bet you're not bottlenecked by energy transfer. Your flow rate is based on how hot you want the water to get, so series or parallel goes back to not mattering.

The equations all work out OK though since one can go substantially above a litre a minute at pressures which are still viable. The standard 18W thing for desktops is about ten times that.

Kind of related from B1M...

>The Paris Olympic Pool is Heated by the Internet

>https://www.youtube.com/watch?v=2gWudPtN6z4&t=4s

I wish I could be as excited as I used to be about Google's infrastructure. But with their constant attacks on the freedom of the internet, they have exhausted my good-will.

So I am afraid I do not care about your liquid cooling on this day. I'm also not particularly impressed, I am sure the devil is in the details at this scale but there doesn't seem to be much novel here.

If a Googler happens to read this and it makes them sad, it's Google, not you. You might consider doing cool stuff somewhere else.

FAANGs have gone from wunderkind darlings that would pave the way for progress, like turn of the 20th-century-style, to monopolistic, feudal overlords rapaciously seeking to exploit land and resources with zero concern for the environment or locals' quality of life.

I see frequent mentions of AI wasting water. Is this one such setup, perhaps with the CDU using the facility's water supply for evaporative cooling?

The CDU is inside the datacenter and strictly liquid to liquid exchange. It transfers heat from the rack block's coolant to the facility coolant. The facility then provides outdoor heat exchange for the facility coolant, which is sometimes accomplished using open-loop evaporative cooling (spraying down the cooling towers). All datacenters have some form of facility cooling, whether there's a CDU and local water cooling or not, so it's not particularly relevant.

The whole AI-water conversation is sort of tiring, since water just moves to more or less efficient parts or locations in the water cycle - I think a "total runtime energy consumption" metric would be much more useful if it were possible to accurately price in water-related externalities (ie - is a massive amount of energy spent moving water because a datacenter evaporates it? or is it no big deal?). And the whole thing really just shows how inefficient and inaccurately priced the market for water is, especially in the US where water rights, price, and the actual utility of water in a given location are often shockingly uncorrelated.

Slightly old but interesting https://blog.equinix.com/blog/2024/09/19/how-data-centers-us...

Even without AI the whole water conservation topic is kinda silly.

The margin at which it makes sense to save water varies wildly by location, but the cultural dominance of the western USA infiltrates everything.

Here in Zürich, my office briefly installed water saving taps. This is a city of less than half a million where the government maintains 1,200 fountains that spew drinkable water 24/7. But someone at the California HQ said saving water is important and someone at the Swiss subsidiary said yes sir we'll look into it right away sir.

It's on the same level as people using incandescent light bulbs. Well we clear 160k Euros after taxes and have public medical care, and electricity is 10c/kWh here, so why does it matter what bulbs we use?

We live in an area surrounded by grass fed cows, so what does it matter if we throw away 3/4 of our steak?

Without regard to how plentiful resources are in our particular area, being needlessly wasteful is in bad taste more than anything. It's a lack of appreciation of the value of what we have.

For water specifically - it is generally speaking the most valuable resource available, we just don't appreciate it because we happen to have a lot of it.

While I'm not saying waste when things are plentiful in general is okay, I think water is a unique case that can be treated differently.

Comparing to energy costs isn't the same because using the energy for the incandescent bulb consumes that energy permanently. The gas/coal/fuel can't be un-burned. Although solar changes this as the marginal cost of that energy is free.

Comparing to food is similar. Once the food is wasted it is gone.

Water is typically not destroyed, it's just moved around in the water cycle. Water consumption in a region is dictated by the throughput the water cycle replenishes the reservoirs you're pulling from. "Waste" with water is highly geographic, and it's pretty reasonable to take exception to California projecting their problems to geographic regions that they aren't important.

It still takes energy, chemicals and therefore pollution to clean water once it's been turned into greywater.

Sun is continuously running a very nice distillation cycle the size of the world that makes fairly clean water just fall out of the sky. It's only a question of where does it fall, and how much. If you want it even cleaner, wait a couple centuries for it to filter down underground, and get it from there - besides maybe a bit high mineral contents, that can easily be removed, it's essentially free, clean water. The only question is how much it's replenished in the area you're taking it from.

There's plenty of areas where there's more rainfall, than there is outflow/evaporation, with water continuously replenishing deep groundwater. "Saving water" in such areas is of little concern besides the basic, economic one of well maintenance - each one can only pull so much, and more usage means more wells, and more upkeep.

> For water specifically - it is generally speaking the most valuable resource available, we just don't appreciate it because we happen to have a lot of it.

And for water specifically, the second order effects from "water saving" programs can be actually negative. Not enough water means that sewers don't work properly any more, leading from events of stink to helping fatbergs grow [1].

To make it worse, the "obvious" idea of scaling down sewer mains doesn't work either because the sewers are (at least in Europe) also used as storm drains, so if you'd scale down the sewers you'd get streets flooded.

[1] https://www.wiwo.de/technologie/blockierte-kanalisation-die-...

Also, water is not readibly fungible as it’s not generally economic to ship large distances. So either you have it, or you dont.

If you do have it, often you have more than you can store or use effectively (the rest just runs off).

If you have a lot of water, then going to a bunch of effort saving it is indeed silly.

If you don’t have a lot of water, then it is indeed essential.

It's nothing like either of those things because both of those things have harms to other people and those harms scale linearly with the amount of consumption. Wasting steak isn't problematic because you run out of cows it's problematic because of the climate impact of raising them.

Saying that saving water is "about respect" or something is idiotic. Saving water is about ensuring there's enough water to go around. This is something you need to do in places where water is scarce and not where it isn't. And if you waste time and energy on saving water you are ultimately making the world poorer.

Obviously I'm simplifying things by talking in absolutes here, but what I said above about "the margin at which it makes sense" gets at the truth of the matter. Installing water-saving taps in Zürich is almost certainly a net harm to the environment.

This is one of those cases where we fly by and don't think much about it because we live in a plentiful environment. The more detailed we get, the more we realize that everything has a cost, and wasting water is not free as in beer. Have we also considered the disposal costs of wastewater?

I used to live in a place where water was infinite. Fast forward 20 years, now it's not anymore, the fish bearing watersheds ultimately bear the price, but everyone is still unmetered and there isn't low flow anything. If you piss away precious resources for no good reason and claim it's not wasteful, shame on you.

Yes, I have considered the disposal costs of wastewater! Yes, I still think importing hundreds of stainless steel widgets from China and then having a plumber spend who knows how many hours installing them is almost certainly a net negative. If you piss away precious resources doing that then shame on you.

I have encountered a lot of references to AI using water, but with scant details. Is it using water in the same way a car uses a road? The road remains largely unchanged?

The implication is clear that it is a waste, but I feel like if they had the data so support that, it wouldn't be left for the reader to infer.

I can see two models where you could say water is consumed. Either talking about drinkable water rendered undrinkable, or turning water into something else where it is not practically recaptured. Tuning it into steam, sequestering it in some sludge etc.

Are these things happening? If it is happening, is it bad? Why?

I'd love to see answers on this, because I have seen the figures used like a kudgel without specifying what the numbers actually refer to. It's frustrating as hell.

The water is "used" in the sense that it evaporates. At a global average rate of 1 liter per kilowatt-hour of energy, Google claims.

Does this evaporated water stay in the loop and condense out later? Is an extra liter on the front end needed in the line until the first liter falls out of the back end?

What does that mean for environmental impact? Hydroelectric power uses water in the sense that the "used" water is at a lower altitude. What happens once that is done, is what matters.

Depending on what global average means, it seems like that's quite a lot of cycling of evaporation unless they are releasing steam at 800C

Looking up overall water usage, the US uses 27.4Billion gallons a day residential and 18.2 Billion gallons industrial. It surprised me that industrial was lower, but I guess the US manufactures less these days.

If the 1kwh per litre were accurate then judging by this calc https://www.google.com/search?q=(27.4billion+gallons+*365)*+... 37 857 903.3 terawatt hours

0.1% of The residential water use of the US would be enough to cool the entire Electicity output of the world (about 30,000TWh)

(of course, with these things it's easy to slip an order of magnitude(or several) so best check my numbers)

It means different things in different places, depending on where the water was sourced and the counterfactual of what would otherwise have happened to it. For example if you source surface water that evaporates, and the local rainshed feeds that watershed, then it's virtually closed-loop. If you're pumping fossil water out of an aquifer, and it rains over the ocean, then it's the opposite.

I suppose in some desert-ish climate the extra moisture given to the air doesn't rain down nearby but moves with the wind somewhere far away where it rains. So there might be an effect of using locally precious water even if it's a closed loop on a continent level.

This article will help you. https://www.construction-physics.com/p/how-does-the-us-use-w...

> ...actual water consumed by data centers is around 66 million gallons per day. By 2028, that’s estimated to rise by two to four times. This is a large amount of water when compared to the amount of water homes use, but it's not particularly large when compared to other large-scale industrial uses. 66 million gallons per day is about 6% of the water used by US golf courses, and it's about 3% of the water used to grow cotton in 2023.

> I have encountered a lot of references to AI using water, but with scant details.

Here you go: https://en.m.wikipedia.org/wiki/Cooling_tower

Data centers use evaporative cooling towers which evaporate water to reject heat to the atmosphere.

My understanding is that most new data centers don't use evaporation cooling these days, at least in water-sensitive areas. Hard to find solid data on this either way, though.

Of course, if you're using dry cooling, it uses more electricity, so hopefully you're using solar, not a source that uses evaporative cooling to produce electricity (if in a dry climate).

AWS had a similar article a couple months ago:

https://www.aboutamazon.com/news/aws/aws-liquid-cooling-data...

In either case I cannot find out how they dump the heat from the output water before recycling it. That's a problem I find far more interesting.

Yes see: https://www.austinchronicle.com/news/2025-07-25/texas-is-sti...

I wonder what the economics of water cooling really is.

Is it because chips are getting more expensive, so it is more economical to run them faster by liquid cooling them?

Or is it data center footprint is more expensive, so denser liquid cooling makes more sense?

Or is it that wiring distances (1ft = 1nanosecond) make dense computing faster and more efficient?

Not sure about classical computing demands, but I think wiring distances definitely matter for TPU-like memory heavy computation.

It's more of a testament to inefficiency, with rising TDP year after year as losses get larger with smaller nm processes. It's so atrocious, even in the consumer sector Nvidia can't even design a connector that doesn't melt during normal usage because their power draw has become beyond absurd.

People don't really complain about crappy shovels during a gold rush though unfortunately, they're just happy they got one before they ran out. They have no incentive to innovate in efficiency while the performance line keeps going up.

It's a mixture of both 2 and 3. The chips are getting hotter because they're compacting more stuff in a small space and throwing more power into them. At the same time, powering all those fans that cool the computers takes a lot of power (when you have racks and racks those small fans add up quickly) and that heat is then blown into hot isles that need to then circulate the heat to A/C units. With liquid cooling they're able to save costs due to lower electricity usage and having direct liquid to liquid cool as apposed to chip->air->AC->liquid. ServeTheHome did a write up on it last year, https://www.servethehome.com/estimating-the-power-consumptio...

I've never done DC ops, but I bet fan failure is a factor too— basically there'd be a benefit to centralizing all the cooling for N racks in 2-3 large redundant pumps rather than having each node bringing its own battalion of fans that are all going to individually fail in a bell curve centered on 30k hours of operation, with each failure knocking out the system and requiring hands-on maintenance.

A cool (ha ha!) solution was the old Cray XT3/4 supercomputers, which were air cooled. But instead of a battalion of tiny fans, each cabinet had a single huge fan at the bottom, blowing air vertically through the cabinet (the boards were mounted vertically). No redundancy, sure, but AFAIU it was reliable enough to not be a problem in practice.

That’s a similar design principle to the Mac Pro trashcan, I guess, which also pulled air through a central column alongside vertical PCBs/heatsinks.

It’s because the chips are networked with very high bitrates and need to be physically densely packed together.

> Or is it that wiring distances (1ft = 1nanosecond) make dense computing faster and more efficient?

Contrary to other posters, I'd argue this effect is relatively small. A really good interconnect fabric might give you ping-pong times on the order of 1 microsecond, which is still 1000 times larger than a nanosecond. Most of the delay will be in the switches and the end nodes, not in the signal traveling over the wire or fiber. Say for a large-ish cluster with a diameter of, say, 100 feet (something like 7 rows of racks, each row 100 feet long, give or take), if liquid cooling allows you to double the density, you could condense it to a diameter of 100/sqrt(2) = 70 ft (about 5 rows of 70 ft each). As a ping-pong involves a signal going both ways, the worst-case increase in signal delay would be (100-70)*2 = 60 ft or 60 nanoseconds (in reality somewhat more since cables have to be routed). So about a 6% increase if we assume the baseline is 1 microsecond. Measurable, yes, but likely very small effect on application performance vs. a ping-pong microbenchmark.

Now where it can matter is that by packing the components more closely together, you can connect more chips via backplane and/or copper connectors vs. having to use optics.

It's also the fact that a large portion of the power used by a data centre is actually spent on cooling, so anything that will make that more efficient is a cost saving.

Seeing in "installation & commissioning" mention of "Leak testing" reminds me of the PowerMac G5 liquid cooling leaks:

https://www.xlr8yourmac.com/systems/G5_CoolantLeak_Repair/G5...

Between 2006 and 2012 I was in DCs. I didn’t live there but I had to visit at stupid o’clock too often.

It’s not a hospitable place.

I would have loved it if cooling was quieter and less extreme.

The reason interfaces are on the back is that’s the intake side. So, bring a sweater.

Or just go to the other side of the rack to warm your hands back up.

That's pretty much the way it had to go, other than doing full immersion cooling like GRC that never caught on. Meta has various rack-based chill water heat-exchanger cooling solutions but this looks a bit more integrated.

I’ve always wondered: what if we could actually use this heat? For example, could we take the heated water from cooling a data center and pump it to places that need it (Greenhouses, Industrial processes, Desalination and water treatment)?

Happens for urban data centers/internet exchanges:

www.aboutamazon.com/news/sustainability/the-super-efficient-heat-source-hidden-below-amazons-seattle-headquarters

It's done sparingly. If you search for "data center waste heat uses" you'll find cases like you mention [1] [2] but the ROI on implementing it seems to be lower than renewable energy infrastructure.

[1] https://www.informationweek.com/sustainability/reusing-waste... [2] https://www.techspot.com/news/97995-data-center-uses-waste-h...

What I don't understand is why not use what I've seen in industrial processes? Why not a low boiling point liquid and a heat exchanger? Or even better power a turbine like biogas plants do

OVH has been doing liquid cooling for quite a while, and is now using immersion also. Somewhat surprising to see at that dirt cheap end of the market.

How much moat Google has in the AI race due to its in-house expertise in data centers and in-house IPR in TPUs? Can any company on Earth match them?

Not enough to prevent open source models making it all useless, I think.

> In theory, a PC builder could try exchanging heat to a toilet tank, allowing for highly efficient cooling with each flush.

The future is here.

In an interesting way, the reverse is also here: some district heating plants use heat pumps to extract waste heat from sewage to provide efficient heat for district heating systems.

See https://www.eneco.nl/en/about-us/what-we-do/sustainable-reso... for a basic overview.

I dream of a future where there's an efficient energy recapture in the house. We waste a lot of heat in the shower, and washing clothes and dishes. We also run the fridge all winter when it's less than 5C outside, and the fridge then produces a lot of heat. And when we're done cooking, the oven is still really hot. Could the home heating system of the future use all this heat (and cold) more efficiently?

I don't see this happening until we have ordinary people living in orbit.

The heat in the oven will slowly leak out into the rest of the house, so the home heating system of the right now already uses that. The fridge example is much more relevant though. If you could get the hot-side radiator part of the fridge thermally-connected with the outside temperature, then it would run much more efficiently. However, you'd lose that heat from the house, so if you're having to heat the house as well it might not be worth it.

The cost of the plumbing would probably exceed the cost of the fridge.

Agreed.

But what if I don't want to heat up the home? Where I currently live I suffer from too good isolation where the kitchen heat builds up too much. I don't need the oven heat and would much rather it be piped out to the water heater.

>"Google extensively validates components with leak testing, uses alerting systems to discover problems like leaks, and takes preventative measures like scheduled maintenance and filtration. They also have a clear set of protocols to respond to alerts and issues, allowing a large workforce to respond to problems in a consistent fashion. It’s a far cry from the ad-hoc measures enthusiasts take to maintain their water cooling setups."

How about the author compare it to MS, Meta, or AWS instead of Joe Blow buying parts online? I would hope that Google had extensive validation and clear protocols. [Roll-Eyes]

My guess would be the average reader is probably far more familiar with people water cooling their own machines than the intricacies of how various large companies are doing water cooling (or not) in their data centers.

Relevant.

OVH has been using liquid cooling for many years:

https://blog.ovhcloud.com/new-hybrid-immersion-liquid-coolin...

https://www.youtube.com/watch?v=S_DHJcQpumI

And yet their claimed PUE is 1.26 which is pretty bad. One way to characterize that overall PUE figure is they waste 3x as much on overhead as Google (claimed 1.09 global PUE) or Meta (1.08).

10 years ago, OVH datacenter SBG1 was below 1.1 with a mixture of natural convection + liquid cooling. But I have not been following in the latest years.

Your linked articles are about immersion cooling, which is "liquid cooling," I suppose, but a very different thing. Do OVH actually use immersion cooling in production? This seems like a "labs" product that hasn't been fully baked yet.

OVH _do_ definitely use traditional water-loop/water-block liquid cooling like Google are presenting, described here: https://blog.ovhcloud.com/understanding-ovhclouds-data-centr... and visually in https://www.youtube.com/watch?v=RFzirpvTiOo , but while it looks very similar architecturally their setup seems almost comically inefficient compared to Google's according to their PUE disclosures.

Based on how their data centers burn down, they don't even use liquid in their fire suppression systems.

Reply on news.ycombinator.com