Project Aardvark: reimagining AI weather prediction
turing.ac.uk
turing.ac.uk
Is there a big Clearinghouse for this data?
Kind of like how fintech algos can be run against historic stock market data to evaluate them.
In France for instance Météo-France has released all of its historical data in January 2024: https://www.data.gouv.fr/fr/organizations/meteo-france/#/dat...
For example, the UK's metoffice has a low resolution medium term global model: https://www.metoffice.gov.uk/research/approach/modelling-sys...
Many commercial and even third party governments rely on the data from NOAA and its archives, on top of that the EU runs its own EUMETSAT fleet of data, plus a ton of national services - unfortunately, the result is there's a looooot of datasets.
NOAA's dataset is public domain [1], EUMETSAT only requires attribution for most of its data [2]. On top of that you got the EU's Climate Data store [3], ECMWF [4], and ECA&D [5].
The service that many private weather services provide is to aggregate and weigh all of the publicly available datasets, and some also add in data from their own ground stations, commercially licensed "realtime" data from governmental services, and their own models as well.
The interesting question is what DOGE will do regarding NOAA - it is increasingly possible that NOAA will shut down, either to be turned into a pay-for-play model, replaced by private services, or just carelessly dropped in its entirety.
[1] https://www.ncei.noaa.gov/archive
[2] https://user.eumetsat.int/resources/user-guides/data-registr...
[3] https://cds.climate.copernicus.eu/
Yes, and more people need to understand this. Too many people seem to think that commercial services won’t be impacted if NOAA stops doing what they do.
I sure hope those “smart” people are capable of understanding that.
Well, NOAA can be privatized, sold off to the highest bidder and be fed money from the annual government to provide said critical infrastructure.
In the end, it's always one giant ass grift.
Backtesting, that's called.
I was also thinking about smartphones. They have barometric data, and while it might vary from phone to phone, I'm sure something like a kalman filter + historic data could do something there.
Think about gathering all the data from "stationary" phones, correlate that with weather sat data, and with real "ground truth" weather stations, and then go back 30 - 60 min / a day, and see what comes out.
For anyone else having a TIL moment: It's apparently for vertical position and supposedly sub meter accurate. o_O
The smartwatch series I use explicitly include it because it's essentially the "all in one" version in a series that had smartwatches designed for aircraft use (including military versions where they serve as backup cabin pressure warning, apparently)
But back to device could weather observation, this would require continuous GNSS, devices don't do that (network location would not be good enough).
https://news.ycombinator.com/item?id=22740466
https://www.theverge.com/2015/6/22/8822767/dark-sky-weather-...
This is a lay-person overview which cites some of the in-depth studies: https://ourworldindata.org/weather-forecasts
There's a paper from Norway that tried end-to-end, but their results were not spectacular. That's the aim of many though, including ECMWF. Note that ECMWF already has their AIFS in production, so AI weather prediction is pretty mainstream nowadays.
Google has a local nowcast model that uses raw observations, in production, but that's a different genre of forecasting than the medium-range models of Aardvark.
It's very clear from the MetNet announcement blog[1] that they require HRRR or other NWP output at runtime.
[1]: https://research.google/blog/metnet-3-a-state-of-the-art-neu...
I work on one of them: https://planetarycomputer.microsoft.com/catalog?filter=weath...
The short answer is, "no." There are some projects like the "NNJA-AI" project at Brightband[1] which is attempting to create such a clearing house in order to focus research efforts across the community.
I have a challenge for the model:
Accurate (within 3deg F) weather predictions for the Kansas City metro more than two days out. As of 2024 these were rarely accurate.[0]
[0]https://www.washingtonpost.com/climate-environment/interacti...
Sniff test: ASTM E230 standard tolerances for the venerable Type K thermocouple is +/- 2.2°C or +/- 0.75%, whichever is greater.
Expectations are in need of recalibration if anyone thinks a single number is going to meaningfully achieve that level of accuracy across a volume representing any metro area at any given point in time, let alone two days out.
Do you think 2C global warming is irrelevant because a single thermocouple couldn't accurately measure it?
These sensors (based on thermal resistance) for example have an accuracy of 0.2 °C under typical conditions [1].
[1] https://www.ti.com/lit/ds/symlink/tmp1826.pdf?ts=17427997638...
here's the scenario. I have started a bunch of seeds inside my house. I am waiting for the "last frost" as per the instructions on the seed packets. Now, how do i use a thermometer to tell if the last frost has passed? You need prediction models that can be accurate out at least a few days with temperature, and a 10% error "at freezing" means my plants either live or die, based on that error.
there's no counterargument, here. "oh just cover the new plants" or "just wait longer" don't work, especially with larger gardens or "farms" or "homesteads". Most of us home-gamers just use environmental clues - number of bugs, buds on dogwood or pecan trees (native ones), when other trees flower. But this year i got a lesson that pecan trees get it wrong, too. They are in the process of leafing out and there was an unpredicted, non-forecast cold snap that took the temperature so close to 32.0F (0.00006C) that i think any plant not equipped for that would have died that night. Now, now i'm fairly certain there's no more frost chance, but it's just a guess.
Now imagine i lived anywhere outside of the subtropics, like nebraska or montana and needed to plant food for livestock or whatever.
What i am asking for is what is promised by weather forecasters and the models. If it can't say with any certainty if it's going to freeze, it's completely worthless for this common circumstance. Like most people, i don't care if the forecast is 75F and it's actually 70F or 80F (or 68F), but what i do care about is a forecast for lows of 50F and it ends up being 32.15F. If you were a roofer and you were off by 18 degrees, you'd still be in prison.
I wonder how poorly this thing operates, and whether taking several years of history to look at would help much.
Would some hypothetical future AI just "know" that tomorrow it's going to be 79 with 7 mph winds, without understanding exactly how that knowledge was arrived at?
Up to a point .. and that point is more or less the same as the point where humans can no longer catch a spinning tennis raquet.
We understand the gravitional rainbow arc of the centre of mass, we fail at predicting the low order chaotic spin of tennis raquet mass distributions.
Other butterflies are more unpredictable, and the ones that land on a camels back breaking a dam of precariously balanced rocks are a particular problem.
* https://en.wikipedia.org/wiki/Tennis_racket_theorem
* Dzhanibekov effect demonstration in microgravity: https://www.youtube.com/watch?v=1x5UiwEEvpQ
The point is whether a LLM has any feelings ...
It was an idle thought that was only barely tangentially related to the article in question, and was not meant to be a comment at all on the model in the article or on any current (or even very near future) AI model.
I don't expect anyone would have an answer, given the extreme degree of hypothetical-ness.
We can query an LLM a simple question like “how many ‘r’ are in the word strawberry”, and an LLM with know access to tools will quite confidently, and likely incorrectly, give you an answer. There’s no actual counting happening, and any kind of understanding of the problem, the LLM will just guess an answer based on its training data. But that answer tends to be wrong, because those types of queries don’t make up a large portion of its training set, and if they do, there’s a large body of similar queries with vastly different answers, which ultimately results in confidentiality incorrect outputs.
But provide an LLM tools like Python, and a “chain-of-thought” prompt that allows it to recursively re-prompt itself, while also executing external code and viewing the outputs, and an LLM can easily get the correct answer to query “how many ‘r’ are in the word strawberry”. By simply writing and executing some Python to compute the answer.
Those two approaches to problem solving are strikingly similar to intuitive vs analytical thinking in humans. One approach is driven entirely by pattern matching, and breaks down when dealing with problems that require close attention to specific details, the other is much more accurate, but also slower because directed computation is required.
As for your hypothetical “weather AI”, I think it’s pretty easy to imagine an AGI capable of confidently predicting the weather tomorrow, not be capable of understanding how it computed the prediction, beyond a high level hand wavy explanation. Again, that’s basically what LLM do today, confidently make predictions of the future, with zero understanding of how or why they made those predictions. But you can happily ask an LLM how and why it made a prediction, and it’ll give you a very convincing answer, that will also be a complete and total deception.
Generally no. If I show you a puddle of water, can you tell me what shape was the ice sculpture that it was melted from?
One is Newtonian motion and the other is a complex chaotic system with sparse measurements of ground truth. You can minimize error propagation but it’s a diminishing returns problem, (except in rare cases like for natural disasters where a 6h warning can make a difference).
[1] https://link.aps.org/doi/10.1103/PhysRevLett.120.024102
[2] https://link.aps.org/doi/10.1103/PhysRevResearch.5.043252
As a result a human running to catch the ball over some distance (eg during a baseball game) runs along a curved path, not linearly to the point where the ball will drop (which would be evidence of having an intuition of the ball's destination).
Certainly what I was coached to do, what outfielders say they do, and what I see watching the game, is to "read" the ball, run towards where you think the ball is going, and then track the ball on the way. I was and am a shitty outfielder, in part because I never developed a fast-enough intuitive sense of where the ball is going (and because, well, I'm damn slow), but watch the most famous Catch[1] caught on film, and it sure looks like Mays knew right away that ball was hit over his head.
The problem with these is that they don't really work, often even in theory. People do seem to predict at least some aspects of the trajectory, although not necessarily the whole trajectory [2].
[1] https://pubs.aip.org/aapt/ajp/article-abstract/36/10/868/104...
[2] https://royalsocietypublishing.org/doi/10.1098/rsos.241291
> "Sma," the ship said finally, with a hint of what might have been frustration in its voice, "I'm the smartest thing for a hundred light years radius, and by a factor of about a million ... but even I can't predict where a snooker ball's going to end up after more than six collisions." [GCU Arbitrary in "The State of the Art"]
The error scales up exponentially with the number of (ball-to-ball) collisions.
So if the initial ball position is off by "half a pixel" (=> always non-zero) this gets amplified extremely quickly.
Your intuition about the problem is probably distorted by considering/having experienced (less sensitive) ball/wall collisions.
See: https://www.lesswrong.com/posts/JehyrC6W3YTtdxw6S/a-primer-o...
I think a consciousness with access to a stream of information tends to drown out the noise to see signal, so in those terms, being able to "experience" real-time climate data and "instinctively know" what variable is headed in what direction by filtering out the noise would come naturally.
So, personally, I think the answer is yes. :)
To elaborate a little more - when you think of a typical LLM the answer is definitely no. But, if an AGI is likely comprised of something akin to "many component LLMs", then one part might very well likely have no idea how the information it is receiving was actually determined.
Our brains have MANY substructures in between neuron -> "I", and I think we're going to start seeing/studying a lot of similarities with how our brains are structured at a higher level and where we get real value out of multiple LLM systems working in concert.
https://apnews.com/article/weather-forecasts-worsen-doge-tru...
In a Nature paper in particular, the final layout is typically done by the journal's professional production team, not the authors.
Not all publishers grant permission for authors to upload the peer-reviewed and layouted postprints elsewhere.
I haven’t downloaded it to see what’s in it.
https://en.wikipedia.org/wiki/Turing_Institute
There was a previous Turing Institute in Glasgow doing AI research (meaning, back then rules-based systems, but IIRC my professor was doing some work with them on neural networks), which hit the end of the road in 1994. There was some interesting stuff spun out of there, but it's a whole different institute.
It's recently struggling for relevance.
https://www.ft.com/content/6bfea441-e16c-499a-a887-69f735c29... (https://archive.ph/ujfhb)
I hope they turn it around because the UK need for AI academic coordination/leadership is so high.
Fun documentary at the time. Something about 12 weeks with geeks, and jumping out of windows comes to mind.
Unable to tell if this is an exaggeration or if I'm just missing nuance, how would the model replace the data gathering which is listed as step 1.
[0]: https://www.revistascca.unam.mx/atm/index.php/atm/article/do...
[1]: https://wmo.int/about-wmo/awards/international-meteorologica...
"Observations" is overloaded; it's really any "raw" weather data product, like satellite imagery, surface station measurements, or weather balloon traces. I'm handwaving away a lot of complexity here.
The AardvarkWeather model is a significant new development and paradigm shift - it's one of the first models of a new class which do not require an analysis, and can directly use the "raw" weather data observations that are typically used to perform data assimilation.
~14GB, needs login, works with some federated accounts.
Weather predictions are for specific events and areas, made on the order of days- typically no more than 2-3 weeks into the future.
Climate models predict future averages over large regions on the order of decades.
"rapid climate change" is on the order of "within this century". Whether the climate changes or not doesn't really impact the weather models at all, because the climate's not changing on the same time scale.
Weather prediction: which path typhoons will follow?
Weather models look at current conditions and extrapolate what the next set of conditions will be.
Another way to look at it is that climate change models reflect what we will experience. Weather models reflect the mechanics of humidity, temperature and air flow- given a current state, what is the next likely state. Climate change doesn't change how those mechanics interact. It only guesses what "current state" will look like in the far future.
Over the course of ten years, the average daily temperature goes up by 1 degree F (which is many times faster than reality).
That means that there's basically one day in the entire year that doesn't fit within the existing model, or maybe a handful if some other part of the year is cooler than normal. We see more variation between normal years than we expect to see average increase in both the short and medium terms.
Our weather models predict no more than two or three weeks into the future.
There's literally nothing about climate change that will change how weather systems interact that will occur faster than our weather models will adapt to. Our weather patterns may change drastically, but that's a fundamentally different problem.
Those models don't just "adapt", they're carefully and manually refined over time, slowly incorporating better physical models as they're developed, and better methods of collecting data as it become available. Climate change absolutely changes how weather systems interact faster than our models can adapt, primarily because we can only adapt models to changes after their accuracy degrades, and it becomes possible to start identifying potential weaknesses in the existing approaches.
You're example of seasonal temperature changes vs global average temperature increases, vastly underestimates how complicated weather models are, and how much impact changes in global climate systems (like the North Atlantic Current) can drastically impact local weather conditions. Weather models, like all models, will make assumptions about the behaviours of extremely large systems which aren't expected to change very much, or are very hard to measure in real-time. As those large systems change due to climate change, the assumptions made by weather models will grow increasingly incorrect. But that hard part is figuring our which assumption is now incorrect, not always easy to identity exactly what assumptions have been made, or accurately measure the difference between the assumption and reality.
Thus, if you want your AI model that predicts weather, which was trained on data before that change, to predict weather after that change, you may very well find that it rapidly loses accuracy.
And every year over year change is within error bars of the prior years results, once you account for those initial conditions.
But if your model is an LLM (or close cousin) purely based on previous weather patterns, then it may struggle to accurately predict once the underlying conditions change.
If you train a weather model ML on past weather data, which has reduced variability, and use it to predict future weather, it is possible that it under predicts variability of weather variables (amount of expected rainfall, maximum temperature tomorrow etc). Variance not mean.
I say "it is possible" because only rigorous modeling can tell us the truth. Which is why I asked the question originally. I don't think it can be denied by abstract arguments.
I can hear the keyboard mashing already. YES, I am aware there are problems in both research and publishing.
That does not mean that a HN'ers "did they think of X?" is any more valid. It's like saying that because bridges collapse, we should be allowed people with no relevant experience, knowledge, or training to look at new bridges and say "that bit there doesn't look strong enough to me!"
The GP asked _how_ robust the ML models will be, which is a perfectly good question to ask. Maybe a climate scientist specializing in ML can answer that question.
"It's like saying that because bridges collapse, we should be allowed people with no relevant experience, knowledge, or training to look at new bridges and say "that bit there doesn't look strong enough to me!""
Not a valid comparison. Civil engineering is a field with a lot of known answers. bridge designs rarely rely on cutting edge research.
I have heard the opinion that the models would not learn weather patterns but rather “weather physics”. I am not knowledgeable enough in ML to comment on that though.
Unless people start interacting with weather forecast differently in the future it doesn't seem like something that needs personalization
That represents a huge step-change in need compute for accurate weather models, and opens up the possibility of even more accurate models, if the accuracy of these techniques scales with available compute. If you can get state-of-the-art accuracy with a desktop computer, something that normally requires a super-computer, what happens if you run those techniques on a super-computer?
Or even "almost as accurate as the current best models".
It'll sort first in any alphabetized list. Before GFS, etc.
By the same measure, it's probably smart to name your company or app A-something rather than Z-something.
Reputedly, "Apple Computers" coming ahead of "Atari" (Jobs' ex-employer) in the phonebook was one of the reasons for Apple being Apple.