Machine learning can boost the value of wind energy | Hacker News Reader

Machine learning can boost the value of wind energy

deepmind.com

148 pointsandreamichi32 comments

I have been working in the same field, quite a few years in my own forecasting company and then later on as a side project.

Not wanting to steal the thunder of Deepmind, but I feel should mention my site which is currently beta as in "works for a few electricity power load and supply forecasting scenarios" and quite well so. Also, it's out there, cheap and fast.

https://ausblick.cryptoport.net

Unfortunately, it's only in German but the basic rule is:

- upload an excel (or csv) file

- wait a minute or so (for a typical dataset of say 3 years of hourly data)

- get back the results

There are a few design rules how the excel file should look like, e.g.,

- a date/time series per sheet (it accepts most English and European formats) and as many number series as you want (always use a decimal point instead of comma here)

- one series has to have consecutive missing numbers at the end, this will be the target value

Other than that, it should just work.

Feel free to comment.

PS: The examples might be the best explanation - https://ausblick.cryptoport.net/examples

What prediction algorithm are you using for this? Not sure if it's mentioned elsewhere on the site

No, it's not mentioned. It's currently a mix of very simple techniques

- a number of smart features (usually a few k) depending on the series (using lags, aggregates, curve fits, combinations of features, ...)

- an iterative algorithm that selects features using maximum relevance (~ correlation with the target) / minimum redundancy and adds them to the model

- simple pca and ridge regression (because it's fast)

- a few optimizations of the final model (removing features, selecting a better ridge regression alpha with CV, ...)

The stack is pure Clojure / Clojurescript.

How did you find developing your models in clojure? I would love to switch away from python for ml but it seems to just have the ecosystem.

Actually, I use both but my production models are in Clojure.

I often end up implementing minor things myself using lower level abstractions (e.g., Linear Regressions or PCA with whitening using Matrix libraries) and I check the results and/or try new things using scikit-learn.

So in general, I'd say I do the programming (outputing intermediate CSVs, tests, web service, thread handling, UI, ...) in Clojure(Script), and try other approaches (e.g., other models/parameters/...) in Python.

I'm quite happy with this pipeline but probably to some extent because I really love to understand how things work and nothing pushes you to learn as much as a missing function in your ML library :-)

I have previously built such a tool for retail sales forecasting (based on previous sales). The stack used R + Rshiny (web app). Method FB's `prophet` as it seemed faster and more accounting of holidays and other variations.

I'd also be interested in hearing a bit more about this.

Yes, me as well.

No paper, no code, no comparison to other models - hell, they didn't even label the graphs. This is just Google PR spam.

Yeah, this is just fluff ignoring prior art. ML and all sorts of other fancy modeling is nothing new for power grids.

The optimization for their datacenters used a 3 layer neural network with 6 neurons. I wonder if this is much more complicated than that...

You can still do cutting edge work with 3 small layers, for example most of the advances in robotic control (using neural networks) use networks with similar sizes.

Well, keep an eye out for it here:

https://deepmind.com/research/publications/

Based on the datacenter cooling work, you'd do better to set up a Google Patents alert instead.

> Using a neural network trained on widely available weather forecasts and historical turbine data, we configured the DeepMind system to predict wind power output 36 hours ahead of actual generation.

Interesting. Something I've long wondered about:

- Many wind farms are located in very specific places that have measured high winds and are otherwise good places to locate a wind farm.

- Global climate change is changing local climates too. Are the studies that were done 10-20 years ago to discover ideal wind farm locations no longer as accurate as they used to be?

- How are wind farm locations decided today? Is there is skill in predicting where a wind farm might be optimally located in 5 years rather than where it should be located today? Or if not (since probably not) what are the right risk mitigation actions to take if you are planning a wind farm but unsure about location?

- Since Machine Learning is useful in improving the value of the existing wind power, could it also be useful in this endeavor, finding optimal wind power generation locations?

- Lastly, is this something a person could research at home with open data and code? Just curious. :)

A demonstration of the astonishing value of a relatively small government program: weather observation and forecasting.

Weather forecasting is not a small program, NOAA & NWS are both pretty big, but they're not the largest.

Actually, scratch that. On the scale of government, they're more medium size.

Is this a substantial improvement over existing neural network models? I’ve seen models that were being used to trade renewables on PJM’s market that seemed at least as good as these. And, similar models are currently being used to manage DERs in micro-grids.

I don't mean to downplay these results. I’m not an expert. Just mentioning it in case someone at deepmind sees this and wants to reach out to PJM. I believe those models were created almost 10 years ago.

It is probably not. Their neural network to optimize their warehouse efficiency had 6 neurons. The neural network used here is probably pretty small too

Reminds me of some of the excellent advice in Bret Victor's essay about global warming:

http://worrydream.com/ClimateChange/

The investment is not only in the machinery, but in tools to model, observe, and predict efficiency.

If anyone is interested in this type of work, our team at Amperon is hiring data engineers, data scientists, and front end engineers. We’re funded by SV Angel and Notation Capital and we’ve already got traction in ERCOT (Texas).

More info: https://medium.com/@astanway/introducing-amperon-2cded368284...

I'm not seeing a data engineer role by title or description in the angelist site linked to at the bottom of that post, is that the right spot?

Is remote (Texas) viable?

How exactly does this need machine learning? Naively I would assume that it's somewhat directly correlated with wind speed (and little more, maybe direction) and a very simple model based on that data will give you good predictions.

I think determining the wind speed and direction 36 hours in advance is the tricky bit.

Weather forecasting is difficult problem but it's problem that as far as I know has been worked on intensely for quite a while.

Weather forecasting is based on physics-based models that, so far as I know, are essentially correct with the fundamental barrier to prediction being a combination of random effects and the difficulty of solving the complex equations numerically.

Deep learning involves training a heuristically approximate a system based on past data. It has been used to emulate various human-learning behaviors like recognizing images or recognizing good or bad position in games. Essentially, the visible successes are in "we don't know but can extrapolate roughly from data" (or we "know" but can't easily program it, in the case of image recognition). So I'd be surprised if deep learning do a good or better job in situation where we do know how things work, we already running algorithm which is correct - the situation with weather.

I didn't see any comparison to other simpler methods. I'd like to see how well a time series model works.

I was super excited when I read the headline and saw that deepmind published the article. I was hoping for something along the lines of applying reinforcement learning on the turbine or park level to optimize power output. Finding the optimal pitch angles, better yawing algorithms (yaw misalignemnts are one of the most common causes of underperforming wind turbines), or finding optimal yaw directions for the entire wind park (downwind wake effects can have a negative impact on other turbines)...

Unfortunately this is really only about predicting wind speeds to then forecast power output. IIRC, grid codes in some countries are starting to require 24 hour, 15 minute interval forecasts for renewables? I've also seen this feature being advertised by other companies over the years as well, although I don't recall seeing any numbers on accuracy for any of the models.

> Our hope is that this kind of machine learning approach can strengthen the business case for wind power and drive further adoption of carbon-free energy on electric grids worldwide. Researchers and practitioners across the energy industry are developing novel ideas for how society can make the most of variable power sources like solar and wind. We’re eager to join them in exploring general availability of these cloud-based machine learning strategies.

Would be nice to know a bit more detail? Which power grids was this approach tested on? How does this approach work exactly? Are we talking about DNN on time series? The graphs also really need labels to inform.

>> ... join them in exploring general availability of these cloud-based machine learning strategies.

... is probably the operative part. Possibly uncharitable, but this appears to be ‘thought leadership’ marketing generally supporting the idea of GCP managed services.

Of course it can, but only on the margin as this article suggests. The real potential is in figuring out where to put them, what to put in the spots, and most importantly, who to choose to build them and support them given the aforementioned variables. There may also be some advances to be made in the actual design, but I do not believe machine learning is going to help so much there.

"Statistics are useful for forecasting."

Meteorologists have long used a metric, forecast skill, to evaluate different techniques. It would be good if the researchers used standard benchmarks.

ML is probably much more useful in understanding the demand for power, the reaction to weather and current events. But even still, it seems a stretch.

Perhaps some people are missing the point here?. The post is not about the technical challenge/novel but rather how existing ML techniques can play a part in making reusable energy more efficient.

Reply on news.ycombinator.com