Google's AI weather prediction model is pretty darn good
theverge.com
theverge.com
[1] more about this: https://press.princeton.edu/books/hardcover/9780691249131/ai...
Google says GenCast forecasts will later be available from them too.
Also ECMWF runs a very similar diffusion model, it's not operational but run a couple of times a day with results available on their graph site (and as data files too I guess): https://charts.ecmwf.int/
The results are governed by physics up to some level, but we can't simulate at that fine a level, so there's some inherent aleatoric uncertainty (i.e. noise). And I would generally say that physics-simulation-ML is not moving as fast as say, inference on images or text. For example, if you see a picture of a car, there's very little inherent uncertainty on what the answer is. If you see the world simulator state there's a lot of uncertainty on what happens next.
That all being said, I think is basically the best model out there , and almost certainly the best open model. This is really the culmination of many years of effort getting data and software in place to run such a large scale training job. Very impressive!
I’m not sure that a blanket statement like this is a valid argument in an article that perhaps suggests the opposite is true.
Unless its a captcha.
I've been thinking about this a lot. Many ML people work with what is "closed-domain" data -- the data is essentially complete (image, sound, words, or any kind of embeddings) with no unmeasured variables, so the ML algorithm is essentially trying to learn a function that can predict this.
Unfortunately a lot of "open-domain" data has tons of unmeasured variables that are contextual. Suppose I were to try to predict how a full a parking lot would be over the course of a week. You can gather lots and lots of data, but still never get to a near-perfect level of accuracy because the co-variates that drive how full a parking lot is (unexpected influencer effects on the demand, competitive forces that happen to shift one day, power outages in the other part of town, other irreducible randomness = "aleatoric uncertainty" in technical parlance) aren't in the data (or at least not completely).
Fortunately this isn't a problem in real life because many effects cancel each other out, so we are able to arrive at a good-enough aggregate prediction. But "open-domain" ML problems will never achieve the kind of accuracy that "closed-domain" ML can achieve, even with tons of data. Closed domain ML can assume a degree of regularity that open domain ML can never assume.
https://charts.ecmwf.int/products/graphcast_medium-mslp-wind...
I was watching it during the recent hurricane season, and it did not seem to perform much differently than other models.
Simple systems can be famously unpredictable [1]. Our bodies manage entropy; that should make them complex but predictable. The weather, on the other hand, has no governors or raison d'être.
Arbitrary precision, not arbitrary length. Even "from [a] mathematical viewpoint, given an exact initial condition, we can gain mathematically reliable trajectories of chaotic dynamic systems" to only a "finite...interval" [1]. (This is due to "numerical noises, i.e. truncation and round-off error, where truncation error is determined by numerical algorithms and round-off error is due to the limited precision of numerical data, respectively.")
For a physical system like the weather, uncertainty "mainly comes from limited precision of measurement," though there is also the "inherently uncertain/random property of nature, caused by such as thermal fluctuation, wave-particle duality of de Broglie’s wave, and so on."
[1] https://www.sciencedirect.com/science/article/abs/pii/S10075...
For example, I can calculate the Fibonacci sequence to an arbitrary length but not infinite.
Skim the paper. Numerical noise means you cannot calculate the 3-body problem to an arbitrary length. There is a finite, mathematical limit even with perfect knowledge of initial conditions.
There isn’t any claim that mathematically exact starting values can’t be propagated with arbitrary precision to arbitrary length, and I would claim that this is possible (but not practical due to compute being limited, of course).
But there’s no hard limit of precision and length where a simulation can’t be made if the starting conditions are exact. The point of the paper is that starting conditions are never exact which limits the length you can propagate.
It talks about that. Which is relevant when we're talking about the weather. But it opens by discussing the hard mathematical limits to numerical methods.
> there’s no hard limit of precision and length where a simulation can’t be made if the starting conditions are exact
Wrong.
Read. The. Paper. Numerical methods for chaotic systems are inherently, mathematically uncertain.
Beyond a certain number of steps, adding precision doesn't yield a more precise answer, it just produces a different one. At a certain point, the difference between the different answers you get with more precision covers the entire solution space.
Even with perfect knowledge of initial conditions, numerical noise limits the forecast interval.
Is this a chaos theory prediction? Why 14 days?
Why not? What's so special about 7-14 days? I can see plenty of reasons one might want to predict accurately the weather even a year out, just one example: will the weather support an outdoor bbq event this late in fall next year?
Sure, but there could be a disease outbreak, or a pork meat recall that prevents it anyways. The weather, as a factor, is generally insignificant with respect to loss of life events. To the extent they are we can see those events far enough in advance at 7 to 14 days to compensate correctly.
A better example would be "can we launch a space craft from this launch site next year." Even then, we never pick a launch day, we establish a launch window, because we already know, the weather changes fast enough that it's unlikely to remain identical for several days in a row. So conditions on a single day are effectively meaningless.
The Space Shuttle program got even better with this by feeding in wind data from high atmosphere probes back into the launch vehicle software so it could plan it's maneuvers ahead of the wind so it could reduce vehicle stresses to within tolerable parameters. They went from a 20% launch probability to an 80% launch probability with this system.
I mean.. enjoy your BBQ either way just bring some popup tents.
It's like the super trading algorithms who achieve perfect scores during backtest.
The question is, how does it perform on unknown events.
From the linked article:
> GenCast is a machine learning weather prediction model trained on weather data from 1979 to 2018
and a google blog https://deepmind.google/discover/blog/gencast-predicts-weath...
> To rigorously evaluate GenCast's performance, we trained it on historical weather data up to 2018, and tested it on data from 2019
I am guessing they did not want to set up the data pipeline to run inference in a live setting. But that is what I would need to see to be a true believer.
Still a cool result and article though.
The Google model is probably the best so far but ECMWF's own diffusion model was already on par with ENS and many point-forecast models (graph transformers, not diffusion) outperform state-of-the-art physical models.
What is missing is initialization directly from observations. All the best-performing models initialize from ERA5 or other reconstruction.
And GenCast was tested against and older model which performs worse.
> The ENS system has improved significantly since 2019, according to ECMWF machine learning coordinator Matt Chantry. That makes it difficult to say how well GenCast might perform against ENS today.
And the testing makes it "difficult to say." The obvious conclusion is "run a new set of tests" but they'd rather pay of the verge to publish half truths instead.