Hacker News Reader

Comment by badrequest | Hacker News Reader

At Google Next in April, I attended a talk by the C-level in charge of Google's Datacenters, and they bragged about using machine learning to manage/anticipate their power draw, and the subsequent efficiencies those brought. They also bragged about how much they're trying to make these centers powered by renewables.

When an audience member asked if they would be releasing their models that help them manage power, the speaker quickly changed the topic, which I felt told me all I needed to know about the sincerity of their motivations.

View on HN

The trained models are likely not directly applicable to any other company's data centers. Instead, Google publishes machine learning research. Open access so that everyone can benefit. Here's a paper on Deepmind's reinforcement learning system for datacenter cooling: https://deepmind.com/research/publications/safe-exploration-...

They could have saved Earth and you would find a fault. This is a great initiative, but you're criticizing about not "open sourcing" it. Google open sources plenty of things. The solution maybe heavily coupled to their infrastructure.

And who knows it was probably out of the scope of the presentation and he simply didn't know if they had any plans to publish research or directly open source it.

I doubt Google's being super protective of data center power optimization stuff. Demand optimization sounds like a relatively straightforward software problem for Amazon, Microsoft, etc to solve. Most of the utility would be for the smaller players.

I did a bunch of work on managing/anticipating the power draw. ( https://news.ycombinator.com/item?id=13747481 ). Something nobody talks about is how grossly wasteful & inefficient Hadoop is. There's a bunch of cited papers (https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.58... ) that study power consumption of Hadoop nodes, but nothing on how to actually reduce the consumption.

So here's something you can personally do to make a dent to this problem. Most of the Hadoop jobs that you write will involve some statistical summary over a dataset. Find the total, or the mean, median, 90th quantile, whatever. Writing a Hadoop map-reduce job is the single worst way to do this.Almost always, you can sample say a 1000 points, get a kernel density estimate via Parzen & then use a table. All quantiles, order statistics, functions of order stats...these can be hand computed for several univariate distributions & their location-scale families, and your real-life data can easily be bounded above below by these estimates. So you can get to > 95% accuracy just by hand-calculating. I can go into details if you like, but I suspect most of you already know how to do this.

Another way to get huge efficiency gains over using Hadoop is to intelligently memoize results and compute as-you-go. But that requires making some bespoke architecture. However I contend that at any scale where it actually makes sense to use Hadoop (especially in the cloud era), it will be cheaper in the medium-term of 1 year or so to dedicate developer time to make something more efficient than a big daily pipeline of Hadoop jobs

You're exactly right. Hadoop and it's ecosystem are... a very poor replica of the system whose predecessor the respective Google papers were written about. There's glaring efficiency things like not supporting complex encodings (last I used HDFS, at least, it could only do full replication) or the query engines on top not implementing sampling in their aggregates as a default/out of the box, which the Google tools do.

The good news is that barely anyone at FAANG scale is using Hadoop any more

Google’s datacenter power management relies on software features that nobody outside Google is prepared to accept. Most operators expect to have thermal headroom for short CPU clock rate bursts and they don’t want some central daemon downclocking their machines. From what I’ve seen outside Google, most organizations would get much more benefit from focusing on utilization than trying to replicate Google fan speed control techniques. If you still have UPSs and dual AC-DC power supplies in every box and your PUE is 2.0 you don’t need the thing that allowed Google to go from 1.11 to 1.10.

The models in isolation aren't useful. The knowledge to create models like them is where the value lies, but it's only possible to generate value if you can do this in a way that is broadly applicable... which is where building systems management partnerships come into play.

Yes, you can argue the point you made, but it's not particularly insightful because the specific IP employed to govern power mgmt at Google DCs is only truly applicable to Google DCs.

Release of source-code and licensing of said code is typically a very significant consideration for a company like Google.

Even a "C-level" person in the singular sense most likely does not have the authority to approve such a thing. And if you don't have approval, you probably shouldn't talk or speculate about it in a public forum.

Don't hate the player. Hate the legal-liability game.

You think you can drop a model into a new data center with a different architecture, different censors, different workloads, and it will magically save energy?

That is a fair point, though I still think if your genuine motivation was the well-being of the Earth, then you'd, I dunno, _try?_

Presuming other people’s motivations from their behavior is a bit of a fool’s errand.

It’s very easy to be wrong, and it’s the worst kind of wrong because you feel so justified. You have reasons.

That I agree with: they should have shared their methodology for saving energy.

They did, the papers behind their tech are published and available free of charge (someone else here linked to https://deepmind.com/research/publications/safe-exploration-... )

even if they didn't, seeing the limits of those models in different environments would certainly help the community overall, and if it saves a tenth of the energy it does for Google somewhere else it would be an enormous immediate reduction in energy consumption on a global scale, so it's hard to see what the downside would be.

If one was truly concerned about 'democratizing AI', as companies like Google so often claim, then sharing access to the trained models would arguably be far more effective than just sharing research papers which many companies don't even know how to implement. In fact the large majority of companies that are responsible for energy consumption don't have any data on the scale that Google has, so I would go as far as call this a bad faith deflection from the beginning.

It's exactly because the majority of companies don't know how to implement those models that it wouldn't do any good. This is the "give me teh codez" of AI. Google's models are specific to their environment, and they would be more likely to generate waste (at least in time and money required to implement them) than to reduce it.

Reply on news.ycombinator.com