Is there a big Clearinghouse for this data?
Kind of like how fintech algos can be run against historic stock market data to evaluate them.
Many commercial and even third party governments rely on the data from NOAA and its archives, on top of that the EU runs its own EUMETSAT fleet of data, plus a ton of national services - unfortunately, the result is there's a looooot of datasets.
NOAA's dataset is public domain [1], EUMETSAT only requires attribution for most of its data [2]. On top of that you got the EU's Climate Data store [3], ECMWF [4], and ECA&D [5].
The service that many private weather services provide is to aggregate and weigh all of the publicly available datasets, and some also add in data from their own ground stations, commercially licensed "realtime" data from governmental services, and their own models as well.
The interesting question is what DOGE will do regarding NOAA - it is increasingly possible that NOAA will shut down, either to be turned into a pay-for-play model, replaced by private services, or just carelessly dropped in its entirety.
[1] https://www.ncei.noaa.gov/archive
[2] https://user.eumetsat.int/resources/user-guides/data-registr...
[3] https://cds.climate.copernicus.eu/
Yes, and more people need to understand this. Too many people seem to think that commercial services won’t be impacted if NOAA stops doing what they do.
I sure hope those “smart” people are capable of understanding that.
Well, NOAA can be privatized, sold off to the highest bidder and be fed money from the annual government to provide said critical infrastructure.
In the end, it's always one giant ass grift.
In France for instance Météo-France has released all of its historical data in January 2024: https://www.data.gouv.fr/fr/organizations/meteo-france/#/dat...
For example, the UK's metoffice has a low resolution medium term global model: https://www.metoffice.gov.uk/research/approach/modelling-sys...
Backtesting, that's called.
I was also thinking about smartphones. They have barometric data, and while it might vary from phone to phone, I'm sure something like a kalman filter + historic data could do something there.
Think about gathering all the data from "stationary" phones, correlate that with weather sat data, and with real "ground truth" weather stations, and then go back 30 - 60 min / a day, and see what comes out.
For anyone else having a TIL moment: It's apparently for vertical position and supposedly sub meter accurate. o_O
The smartwatch series I use explicitly include it because it's essentially the "all in one" version in a series that had smartwatches designed for aircraft use (including military versions where they serve as backup cabin pressure warning, apparently)
But back to device could weather observation, this would require continuous GNSS, devices don't do that (network location would not be good enough).
https://news.ycombinator.com/item?id=22740466
https://www.theverge.com/2015/6/22/8822767/dark-sky-weather-...
The short answer is, "no." There are some projects like the "NNJA-AI" project at Brightband[1] which is attempting to create such a clearing house in order to focus research efforts across the community.
This is a lay-person overview which cites some of the in-depth studies: https://ourworldindata.org/weather-forecasts
There's a paper from Norway that tried end-to-end, but their results were not spectacular. That's the aim of many though, including ECMWF. Note that ECMWF already has their AIFS in production, so AI weather prediction is pretty mainstream nowadays.
Google has a local nowcast model that uses raw observations, in production, but that's a different genre of forecasting than the medium-range models of Aardvark.
It's very clear from the MetNet announcement blog[1] that they require HRRR or other NWP output at runtime.
[1]: https://research.google/blog/metnet-3-a-state-of-the-art-neu...
I work on one of them: https://planetarycomputer.microsoft.com/catalog?filter=weath...