Hacker News Reader

Comment by brutusborn | Hacker News Reader

I loved this part at the end: "By contrast, allowing a temperature increase for chips in large data centers could have surprising environmental benefits. To this point, Keysight’s White recalled that a company once requested JEDEC increase the spec for an operating temperature by five degrees. The estimate of the potential savings was stunning. Based on how much energy they consumed annually for cooling, they calculated a five degree change could translate to shutting down three coal power plants per year. JEDEC ultimately compromised on the suggestion."

View on HN

I've heard of some large companies that run their data centers hot.

Cheaper to have a slightly higher failure rate, or have the computers throttle their clock speed, than to pay for extra air conditioning.

Hard to do the math here though because you don't know the failure statistics in advance. Kind of a multi-armed bandit problem of sorts.

Wouldn't the Arrhenius equation[1] be a good approximation? It's used in the industry[2] from what I know.

Of course you'll need some data for calibrating the model, but if you got that?

[1]: https://en.wikipedia.org/wiki/Arrhenius_equation

[2]: https://www.ti.com/lit/an/snva509a/snva509a.pdf

Well, sometimes, and more frequently now at really small process nodes, but mostly no. Most electronic failures are basically mechanical and thermal cycling will cause fatigue failures more than elevated temperatures will accelerate things like electromigration. Lots of people still use 1960s style handbook methods anyway because there's no plug-and-chug replacement.

The groundbreaking work here was by Michael Pecht back in the early nineties: https://apps.dtic.mil/sti/pdfs/ADA275029.pdf

Could also be more likely that it's cheaper to extend the life of an older DC by accepting higher temperatures and failure rates than to upgrade the hvac to accommodate newer higher density designs

Sounds like a win/win incentives alignment.

That creates a very unreliable DC though. I guess you'd factor the increase whole-of-facility downtime into your forecasts and work around that maybe.

The air conditioning is not free.

Downtime isn't free.

Depending on design, if the DC isn't very monolithic, then you can run with components failing and not causing much trouble, replacing when available, if your apps are designed to run on a zillion small CPUs, as I believe Google and Amazon are, instead of a giant monolithic computer, 1970s style.

Could have heavier AC for critical components, the main routers and disk farms, for instance.

Details of the calculations unknown to me.

Only three? That's not an immediate win. What if the temperature increase causes just so slightly more failures, causing so slightly more replacements, and each replacement requires energy to make, ship, install, replace, recycle, the effects of increased demand... What if it lasts slightly less long, causing more early failures and eWaste? After all that potential risk, is it a benefit still, and if so, how much?

We don't know and it is hard to know - but I don't blame JEDEC and would not call it a "compromise" on their part like it was a superior option.

Cmon.. logically it has to be more than just three right? "Cooling efficiency" sometimes come in unit of W/°C difference, so I'd imagine that a few more degrees would be a huge deal.

I don't know how much power efficiency would be saved - my concern is more that it is completely logical that running any part at higher temperatures causes increased risk of failure, whether it be a computer part or a mechanical part. How much? I don't know - I just don't blame JEDEC for recognizing this is not a clear and obvious win.

Imagine if the failure rate was raised by as little as 1%. RAM Failure is not uncommon compared to other parts - I've had it happen before and render a system unable to boot, that's why we have Memtest86 and not CPUtest86 or SSDtest86. A 1% increase in failure over 5 years would have effects just as unbelievable as the power saving that increasing the temperature would be. How many smartphones would be junked? How many PCs would be thrown out for not working by people who are average Joes who can't diagnose them, and the extra waste that generates from both disposing the old PC and purchasing a new one? Perhaps the new PC is more efficient, but which is better, greater emissions or more eWaste in the ground due to the new PC being likely more efficient than the old one?

The point is that it is not a clear win. With further research it might be, and I might be all for it. I'm only nitpicking the description of it as being a "compromise" as though it were obvious.

[@picture: I'm at my posting limit for the day because HN is, well... I'll leave their censorship policies for another day. I would agree with you if the RAM with the 90C limit were strictly ECC RAM because that is most often used in data centers and not consumer parts. Maybe we have non-ECC/85 RAM and ECC/90 RAM options...]

That's for sure. I 100% agree that increased temperature will statistically increase failure rate. I'm just thinking that, the most common mechanisms of thermal failure in electronics are caused by repeated thermal cycling which cause fatigue and stress failures at interconnects (solder bumps, silicon bonding, etc). Data centers are designed to be operated in a relatively very constant temperature environment, so I would suspect that the failure rate may not be raised significantly.

Well now you're just being hyperbolic. As you say, this is an engineering problem so solutions are far from their ideals states in either direction. However, 1% increase in ram failure rates ruining the world? That doesn't sound right. Errors are encountered in RAM all the time and guess what, they're corrected often by the hardware before even bothering the system. I'm sure we could deal with a 1% increase...

Most datacenter hardware is fine at 95 degrees F (inlet temp). Approved configurations are usually available to 105 degrees F or slightly higher. Some devices can run as high as 130F.

In the operating range, you’re not going to have any measurable change in operations or failure rate - if you do the parts are defective. All of the stories you hear about this and that are conjecture.

Interestingly, computer chips can often be run at lower voltage and wattage for a given frequency if they are kept at a colder temperature. As a home user I can significantly reduce power draw for a CPU/GPU by improving the cooling solution and lowering voltages.

The reasons this doesn't work for datacenters is two-fold, I think: First, they won't see efficiency improvements just by keeping their CPUs and GPUs (or RAM) cooler because the power levels/tables for the chips are baked-in, and operators aren't going to the trouble of tweaking voltages themselves. Second, even if they did tweak voltages, the cost of sustaining lower temperatures with better cooling likely won't outweigh the savings resulting from lower power draw for the chips.

Still, this raises the question of whether designing hardware for higher operating temperatures is always the right move. At some point there's going to be a cost in performance and/or efficiency that outweighs the savings from allowing higher temperatures. Ideally these tradeoffs should be balanced as a whole.

I think you missed the biggest reason this doesn't do much for servers - they already run at low frequencies & voltages.

For example take the Epyc 7742, at 225W it sounds super power hungry. But the 64-core chip only boosts to 3.4ghz max (2.25ghz base). That's less than the base clock of almost any of the Ryzen consumer CPUs. And if you look at the lower frequency end of https://images.anandtech.com/doci/16214/PerCore-1-5950X.png there's not a whole heck of a lot of efficiency gains likely to be had below that ~3-3.4ghz mark. They're already basically sipping power at something like 3w per CPU core or less. 225w / 64c = 3.5w/c, but the IO uncore isn't exactly cheap to run and iirc sits more like in the 50-70w range. So subtract that out and you're at more like 2.5-2.7w/c. I don't think throwing cooling at this is really going to get you much of a gain.

This usually isn't true at scale though. The chip manufacturers do a ton of validation and qualification to set the operating parameters (voltage, etc).

You can undervolt (or overclock) one specific chip, individuals have been doing this at home for basically ever, but there's (almost) always a system-specific validation process you then do to make sure the system is stable for a specific workload with the new parameters.

And these parameters differ between batches of chips, or even between chips within a batch.

It's also significantly harder to drastically reduce the temperature of the chips inside of a single server, given the machine density of a typical data center.

> It's also significantly harder to drastically reduce the temperature of the chips inside of a single server, given the machine density of a typical data center.

It’d be fun, however, if we could dynamically adjust that according to workload. If workload is light, you could consolidate load into fewer sockets/memory sticks and power down everything in that socket.

It’s been a while since I was doing hardware in a DC, but even dynamic frequency changes in servers were noticeable in workloads.

We’d see serious latency spikes with high volume increases. And high volume would correlate to high I/O on shared storage exacerbating the problem. You’d get a bunch of transactional failures from the spontaneous latency hit.

Same was true with older virtualization/partitioning of hardware. You’d even see apps back up into memory and swap themselves to death.

Now, the apps were awful: inefficient and intolerant. But that’s the reality for the vast majority of business software. Churned out by devs with little understanding and tight timelines for features.

Reliability being an NFR kills me.

It’s an interesting thought. I think in the datacenter today it makes more sense to evacuate and shutdown servers vs. tune down the devices.

You usually have to budget a few °C of difference just for pushing enough energy through heat exchangers. So the rate of chip temperature / external temperature is lower than the rate that effectively determines the cooling efficiency.

One single company changing one single design parameter and enabling savings on the scale of multiple power plants? That is as immediate as wins get.

For one company? That’s pretty impressive.

When I was on an architecture team that consolidated ~80 datacenter to 3 circa 2010, this was a key dollar driver. We raised the temperature ~6 degrees from the average temp, which meant kicking out a few vendors initially. The cost savings for doing that was essentially the total operational costs of 5 datacenters.

The annual failure rates for the hardware did not change at all by any metric. Number of service impacting hardware failures went to zero due to the consolidation.

In general, if you operate within the operating ranges of your hardware, you won’t have failure. You will have complaints from employees, because computers will operate at temperatures not comfortable for humans.

We did the same several years ago. At the time I found http://www.cs.toronto.edu/~bianca/papers/temperature_cam.pdf quite interesting. I didn't find many other precise papers about issues when running at a higher temperatures, but many "should" and "can".

Based on ASHARE there are two guidelines from HPE and IBM: https://www.chiltrix.com/documents/HP-ASHRAE.pdf https://www.ibm.com/downloads/cas/1Q94RPGE

We found (by measurement) that some places in the datacenter with suboptimal airflow are well over the average or simulated temperature so one can leave the safe temperature envelope if one isn't careful.

The beauty of a big project like is you get the engineering resources to lake sure the datacenter is working right.

The hyper scale people take this to the next degree.

It's almost certainly Google, since they've historically ran their data centers hotter than most [0]. Cooling efficiency increases with a higher delta-T to the working fluid, and Google uses a gigawatt or two continuously [1]. From PUE numbers that's hundreds of MW spent on cooling, so making it more efficient is quite worth it.

[0] https://www.google.com/about/datacenters/efficiency/ [1] https://www.cnbc.com/2022/04/13/google-data-center-goal-100p...

Don't you want to have AC running on cooling mode or dehydration mode to get water out of the air?

edit: this makes me wonder what is the ideal humidity in a data centre? Is too dry a thing?

Dry air makes static electricity buildup more.

Too dry is worse than too humid (to a point).

I’m not in this game anymore, but I remember Yahoo built a datacenter in NY off Lake Erie that wasn’t cooled. They aligned the building lengthwise on a ridge above the lake and just blew air through it with no AC. No idea how it operated though.

> The annual failure rates for the hardware did not change at all by any metric.

I don't know what kind of magical hardware you ran, but from the datacenters I've interacted with this is absolute nonsense. Higher temps and/or humidity have an immediate effect on failure rates, to the point where we could plan extra time for them next week if the temp raised even .5 degrees.

Reply on news.ycombinator.com