Hacker News Reader

Comment by theluketaylor | Hacker News Reader

Plus data centers are low hanging fruit for decarbonization. Everything (apart from diesel backup generators) is already electric, so as the grid de-carbonizes so do data centers.

As you mention, DC operators are already highly incentivized to reduce energy costs, so as wind and solar power continue to drop in costs DC operators will want to fund or buy green power as often as possible.

Getting backup power to be carbon neutral will be a little more challenging in a DC, but it's a walk in the park compared to many sectors. Shelf-stable biofuel is already fairly accessible and using grid scale batteries can reduce the need to combust anything (plus open power arbitrage opportunities).

View on HN

I’m not sure datacenters have a great incentive to reduce energy costs (a lesson I learned the hard way after investing in a DC energy optimization software company). People costs are much greater than energy costs. Electricity just doesn’t cost that much in the grand scheme of things. If additional energy spend is necessary to ensure uptime, the choice is an easy one.

Data centers are the cost. I worked for an entity that automated key operational workflows. They went from 15,000 employees in that division to 4,000. The energy cost of those buildings, overhead, etc far exceeded the datacenter, and the datacenter cost and use gets whittled down every year.

> People costs are much greater than energy costs.

Can you prove this assertion? The first page of Google results indicates energy is a greater cost than labor, with figures ranging from 60-70 percent of total operating costs.

(i owned a small web hosting company in the early 00s, and am somewhat familiar with the per sq ft cost model of colos and datacenters in general, having had to contract for space and perform part of the buildout myself)

I think this is taking an expansive view of labor e.g. writing all datacenter applications as optimal code in low level languages.

The general disinterest in performance/efficiency work even for large scale datacenters is a major theme of Dan Luu’s writing. I’m not sure how that applies to the rest of the industry but I can attest that my company’s performance engineering team has been zeroed in every round of layoffs we’ve ever had.

The larger your footprint in the cloud is, the larger incentive you have to make it more efficient though. I think it’s a somewhat self-correcting problem.

The biggest players are running highly, highly efficient software.

Dan Luu’s point is the big players are in fact leaving 7+ figure optimizations on the table all the time, and not really caring when they get found. We’re of course free to disbelieve him, but it tracks with my experience.

The larger you get, the better electricity rates you can negotiate, increasing the opportunity cost of focusing on efficiency.

As long as the industry is growing (it is), the opportunity cost will be high. It seems more likely it's the opposite of a self correcting problem. My 2 cents.

> but I can attest that my company’s performance engineering team has been zeroed in every round of layoffs we’ve ever had.

That doesn't necessarily mean management doesn't care about efficiency and performance, it just means they don't value the performance team (right or wrong).

It's certainly possible for an organization to fail to benefit from performance work. They can make all the servers twice as fast but if all that does is lower the utilization from 10% to 5%, that organization saves nothing, and Amazon pockets the difference in energy usage.

A friend of mine was in management at a large company trying to build up infra to compete with AWS 10-15 years ago (they started this effort when AWS was only S3). This company was bringing in billions a year operating dedicated off-site data centers for other companies, and they saw that AWS was a major threat to their business model. They dumped many, many millions of dollars into building up their infra.

When they gave up, they concluded that AWS's core advantage was their optimization of the electricity needed for cooling. Given that my friend's company failed to design their systems to heavily optimize for this, the cost of electricity for cooling alone didn't let them get near AWS prices without incurring significant losses. He said that if they did it again, every part of the product would have a strict heat budget.

fwiw, I'm at a decently sized hosting company and that assertion is accurate for us.

Maybe your personal experience differs because you were not operating at scale? Larger scale means better negotiating over power / colo costs.

Jouppi showed that TCO is perfectly correlated with energy.

https://gwern.net/doc/ai/scaling/hardware/2021-jouppi.pdf

The actual staffing of DCs is rather sparse for hyperscalers and are typically outsourced if not prime DCs to a few companies that also do it with fairly low headcount.

Hyperscalers in particular currently can’t scale out very easily necessarily because many of the DCs have hit local power restrictions so compute power efficiency and density are the primary means to grow capacity. Think apartments in skyscrapers v row houses. So electricity _efficiency_ is what is being sought rather than just pure operational costs of the DCs.

I’m the northern Virginia exurbs hyperscalers have bought out so much of the land it’s now perhaps greater land use than family farms there and starting to encroach upon housing developments. Locals also complain about the noise (there’s a sort of hum from the sheer mass of air and whirring fans) from DCs impacting health. Some areas have been reporting increased rates of hearing disabilities although I can’t recall a study being conducted yet.

DCs are power hungry and so can have lots of strain in local places.

In popular siting locations like Ireland, the local power grid is struggling. Data centers make up 18% of Irish electricity consumption: https://www.euronews.com/green/2023/06/13/data-centres-gobbl...

In developing countries like India where the power grid is not stable, to maintain DC uptime you probably need backup generation, which is very expensive very fast.

Reply on news.ycombinator.com