HNHacker News
TopNewBestAskShowJobs

meteo-jeff

547 karma · joined September 12, 2021

Passionate about meteorology, data science and software development. Home at https://open-meteo.com/
submissionscomments
meteo-jeff··on GraphCast: AI model for weather forecasting
I would like to see an independent forecast comparison tool similar to Forecast Advisor, which evaluates numerical weather models. However, getting reliable ground truth data on a global scale can be a challenge.

Since Open-Meteo continuously downloads every weather model run, the resulting time series closely resembles assimilated gridded data. GraphCast relies on the same data to initialize each weather model run. By comparing past forecasts to future assimilated data, we can assess how much a weather model deviates from the "truth," eliminating the need for weather station data for comparison. This same principle is also applied to validate GraphCast.

Moreover, storing past weather model runs can enhance forecasts. For instance, if a weather model consistently predicts high temperatures for a specific large-scale weather pattern, a machine learning model (or a simple multilinear regression) can be trained to mitigate such biases. This improvement can be done for a single location with minimal computational effort.

meteo-jeff··on GraphCast: AI model for weather forecasting
Traditional database systems struggle to handle gridded data efficiently. Using PG with time-based indices is memory and storage extensive. It works well for a limited number of locations, but global weather models at 9-12 km resolution have 4 to 6 million grid-cells.

I am exploiting on the homogeneity of gridded data. In a 2D field, calculating the data position for a graphical coordinate is straightforward. Once you add time as a third dimension, you can pick any timestamp at any point on earth. To optimize read speed, all time steps are stored sequentially on disk in a rotated/transposed OLAP cube.

Although the data now consists of millions of floating-point values without accompanying attributes like timestamps or geographical coordinates, the storage requirements are still high. Open-Meteo chunks data into small portions, each covering 10 locations and 2 weeks of data. Each block is individually compressed using an optimized compression scheme.

While this process isn't groundbreaking and is supported by file systems like NetCDF, Zarr, or HDF5, the challenge lies in efficiently working with multiple weather models and updating data with each new weather model run every few hours.

You can find more information here: https://openmeteo.substack.com/i/64601201/how-data-are-store...

meteo-jeff··on GraphCast: AI model for weather forecasting
Not yet, but I am working towards it: https://github.com/open-meteo/open-meteo/issues/206
meteo-jeff··on GraphCast: AI model for weather forecasting
Both APIs use weather models from NOAA GFS and HRRR, providing accurate forecasts in North America. HRRR updates every hour, capturing recent showers and storms in the upcoming hours. PirateWeather gained popularity last year as a replacement for the Dark Sky API when Dark Sky servers were shut down.

With Open-Meteo, I'm working to integrate more weather models, offering access not only to current forecasts but also past data. For Europe and South-East Asia, high-resolution models from 7 different weather services improve forecast accuracy compared to global models. The data covers not only common weather variables like temperature, wind, and precipitation but also includes information on wind at higher altitudes, solar radiation forecasts, and soil properties.

Using custom compression methods, large historical weather datasets like ERA5 are compressed from 20 TB to 4 TB, making them accessible through a time-series API. All data is stored in local files; no database set-up required. If you're interested in creating your own weather API, Docker images are provided, and you can download open data from NOAA GFS or other weather models.

meteo-jeff··on GraphCast: AI model for weather forecasting
Sorry, decades.

KML files for storm tracks are still the best way to go. You could calculate storm tracks yourself for other weather models like DWD ICON, ECMWF IFS or MeteoFrance ARPEGE, but storm tracks based on GFS ensembles are easy to use with sufficient accuracy

meteo-jeff··on GraphCast: AI model for weather forecasting
Extreme weather is predicted by numerical weather models. Correctly representing hurricanes has driven development on the NOAA GFS model for centuries.

Open-Meteo focuses on providing access to weather data for single locations or small areas. If you look at data for coastal areas, forecast and past weather data will show severe winds. Storm tracks or maps are not available, but might be implemented in the future.

meteo-jeff··on GraphCast: AI model for weather forecasting
In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather data.

Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable.

See: https://open-meteo.com

meteo-jeff··on Ask HN: How is your Apple WeatherKit transition going?
I have heard the same regarding 5xx errors in the past couple of months. I am also working on open-source weather API https://open-meteo.com/. It covers most of WeatherKit features and offers more flexibility. You can either use the public API endpoint or even consider to host your own API endpoint.

Forecast quality should be comparable as the API uses open-data weather forecasts from the American weather service NOAA (GFS and HRRR models) with hourly updates. Depending on the region, weather models from other national weather services are used. Those open-data weather models are commonly used among the most popular weather APIs although without any attribution.

If you have any questions, let me know!

meteo-jeff··on Pirate Weather
You can try my API: https://open-meteo.com/en/docs/historical-weather-api
meteo-jeff··on Show HN: Briefsky – a free Dark Sky clone for multiple weather APIs
That’s exactly what I want to change with my open source weather api https://open-meteo.com

It collects raw weather mode data and redistributes weather forecasts with simple APIs

Briefsky is also using it :)

meteo-jeff··on Pirate Weather: A free, open, and documented forecast API
Snowfall is already available in the historical weather API. Because the resolution is fairly limited for long term weather reanalysis data, snow analysis for single mountains slopes/peaks may not be that accurate.

If you only want to analyse the weeks to get the date of last snowfall and how much power might be there, use the forecast API and the "past_days" parameter to get a continuous time-series of past high-resolution weather forecasts.

meteo-jeff··on Pirate Weather: A free, open, and documented forecast API
Hi, creator of open-meteo.com here! I am using a more wide range of weather models to better cover Europe, Northern Africa and Asia. North America is covered as well with GFS+HRRR and even weather models from the Canadian weather service.

In contrast to pirate weather, I am using compressed local files to more easily run API nodes, without getting a huge AWS bill. Compression is especially important for large historical weather datasets like ERA5 or the 10 km version ERA5-Land.

Let me know if you have any questions!

meteo-jeff··on Ask HN: DarkSky Alternatives?
I am working on an open source weather api: https://open-meteo.com

If you are looking for raw weather forecast data, it could be a good start

meteo-jeff··on Historical weather data API for machine learning, free for non-commercial
Thanks for the info. Snowfall was recently added. I am afraid there could be a bug with that particular variable.

Temperature, clouds, etc, seem fine

EDIT: Issue identified and will be fixed in the next days! Thanks!

meteo-jeff··on Historical weather data API for machine learning, free for non-commercial
I have not. It looks promising as it seems to offer multi dimensional data storage and some compression aspects.

I must also admit, that I like my simple approach of just keeping data in compressed local files. With fast SSDs it is super easy to scale and fault tolerant. Nodes can just `rsync` data to keep up to date.

In the past I used InfluxDB, TimescaleDB and ClickHouseDB. They also offer good solutions for time-series data, but add a lot of maintenance overhead.

meteo-jeff··on Historical weather data API for machine learning, free for non-commercial
`300 GB` to `10 GB` was bit over optimistic ;-) 300 GB already included 3 weeks of data. `100 GB` to `10 GB` is a more realistic number.

Many weather variables like precipitation or pressure are very easy to compress. Variables like solar radiation are more dynamic and therefore less efficient to compress.

Getting radar data is horrible... In some countries like the US or Germany, it is easy, but many other countries do not offer open-data radar access. For the time being, I will integrate more open datasets first

meteo-jeff··on Historical weather data API for machine learning, free for non-commercial
Sure. What do you think about "&start_date=20220701" and "&end_date=20220714"?

If end_date is not specified, it would return start_date with 7 days forecast

meteo-jeff··on Historical weather data API for machine learning, free for non-commercial
Yes, it is a combination of delta coding, zigzag, bitpacking and outliner detection.

It only works well for integer compression. For text-based data, results are not use-full.

SIMD and decompression speed is an important aspect. All forecast APIs use the compressed files as well. Previously I was using mmap'ed float16 grids, which were faster, but took significantly more space.

meteo-jeff··on Historical weather data API for machine learning, free for non-commercial
Sure. I bundle a small rectangle of neighbouring locations like 5x5 (= 25 locations). The actual weather model may have a grid like 2878x1441 cells (4 million).

Inside the 5x5 chunk, I subtract all grid-cells from the center grid-cell. The borders will then contain only the difference to the center grid-cell.

Because the values of neighbouring grid-cells are similar, the resulting deltas are very small and better compressible.

meteo-jeff··on Historical weather data API for machine learning, free for non-commercial
Some technical background:

Open-Meteo offers free weather APIs for a while now. Archiving data was not an option, because forecast data alone required 300 GB storage.

In the past couple of weeks, I started to look for fast and efficient compression algorithms like zstd, brotli or lz4. All of them, performed rather poor with time-series weather data.

After a lot of trial and error, I found a couple of pre-processing steps, that improve compression ratio a lot:

1) Scaling data to reasonable values. Temperature has an accuracy of 0.1° at best. I simply round everything to 0.05 instead of keeping the highest possible floating point precision.

2) A temperature time-series increases and decreases by small values. 0.4° warmer, then 0.2° colder. Only storing deltas improves compression performance.

3) Data are highly spatially correlated. If the temperature is rising in one "grid-cell", it is rising in the neighbouring grid cells as well. Simply subtract the time-series from one grid-cell to the next grid-cell. Especially this yielded a large boost.

4) Although zstd performs quite well with this encoded data, other integer compression algorithms have far better compression and decompression speeds. Namely I am using FastPFor.

With that compression approach, an archive became possible. One week of weather forecast data should be around 10 GB compressed. With that, I can easily maintain a very long archive.

meteo-jeff··on Historical weather data API for machine learning, free for non-commercial
Actually, they are historical weather forecasts, but assembled to a continuous time-series.

Storing each weather forecast individually to a performance evaluation for "how good a forecast in 5 days is", would require a lot of storage. Some local weather models update every 6 hours.

But even with a continuous time-series, you can already tell how good or bad a forecast compared to measurements are. Assuming, your measurements are correct ;-)

meteo-jeff··on Historical weather data API for machine learning, free for non-commercial
Hi farmin,

so far I do not offer commercial options, just to keep me out of any potential legal issues. In the next weeks I will review everything, make sure attributions and licenses are correct, and remove the non-commercial limitation.

Australia is currently only covered by a global weather model from German Weather service DWD. I will check if BOM offers some open-data models

meteo-jeff··on Historical weather data API for machine learning, free for non-commercial
Hi bernulli,

1) Data are coming from multiple weather models. Primary data source is the German Weather service DWD with the ICON weather model. In my past experiences, the DWD ICON model performs best for many regions. DWD ICON has a global (~13 km), European (7 km) and a Central Europe (1-2 km) "domain". A higher resolution can improve forecast accuracy, but this is not guaranteed.

For Open-Meteo APIs, multiple models are mixed together. Typically high resolution domains only provide 3-5 days of forecast, afterwards they are combined with a global model.

For North American locations, I am going to add high resolution domains from NOAA as-well.

2) For now, only couple of months archive are available. There will be no limit of how much data can be stored. Data is fairly well compressed while still maintaining good read performance.

I am working on a long term archive as well. ECMWF provides a reanalysis dataset called ERA5 [1] with data from 1959. It will still take me a couple of weeks to process it. With 23 weather variables, it requires around 20 TB disk space (Gridded float32 with deflate compression).

[1] https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysi...

meteo-jeff··on Updated WHO global air quality guidelines
Thanks for posting! This is a good reference to correctly implement and document APIs for Open-Meteo [0].

I soon want to integrate CAMS data from the Copernicus Project to offer Air Quality APIs for non-commercial use applications.

[0] https://open-meteo.com/

meteo-jeff··on Home Assistant – open-source home automation
Hi balloob! I was showing my non-commercial weather API project [0] to HN 2 days ago. Another HN member was already pointing out your project.

If you are interested, we could have a look at integrating weather data without the need of your users to sign up for API plans with their personal data.

[0] https://open-meteo.com

meteo-jeff··on Show HN: Weather API for non-commercial use
Great thanks! Let me know if you are missing something to integrate it.

My goal is to keep Open-Meteo free for non-commercial use and not start introducing API keys with different tiers of payments! The uptake in the recent days is great and motivates me to keep working on it!

meteo-jeff··on Show HN: Weather API for non-commercial use
Nice app. Even open source :)

Regular "shortwave_radiation" should fit your needs. This is basically "direct_radiation + diffuse_radiation".

I am not entirely sure, if this effects your users that much, because many will work indoors and only a fraction of sunlight will get in.

You can contact me directly via mail for a non-commercial use waver.

meteo-jeff··on Show HN: Weather API for non-commercial use
For the VMs, the absolut minimum memory requirement is 16GB. The more, the better.

Otherwise it takes around 40 GB of disk space for every day of data. For an history of 100 days, 4000 GB are required. With compression I could save 50%, but have to invest a couple of days development time to make is work. You could calculate the AWS bill now ;-)

Data on cold storage is an option, but it also super slow....

Currently, I did not yet integrate all the high-resolution models that I want to. Coverage for Europe is great. In North America I will add high resolution NOAA models next.

Most likely I will keep only a limited subset of data as history, but on fast storage to make is accessible quickly

meteo-jeff··on Show HN: Weather API for non-commercial use
I was thinking to use the geonames location database and build a simple fuzzy search engine around it.

What wikidata objects are you referring to?

meteo-jeff··on Show HN: Weather API for non-commercial use
Sure, what kind of fire weather metrics could be added? Can they be derived from soil moisture, temperature, wind, humidity? Or are you referring to gridded fire indices from a different source?

For now I would refrain to include data that is non-date or has restrictive licences.

You can find my email on the open-meteo site and drop me some information

Page 1 of 2Next →