meteo-jeff

Glad you like it! Bootstrap v5 helped a lot :)

I hope to improve the UX a bit further with a city-search for example. Unfortunately I am lacking a good geocoding API for that.

Correct. This cannot be prevented.

Also for API keys you can just search for keys from basically any weather API and you will find credentials on GitHub. People also sign up 10x for free API services to circumvent typical request limits per day.

At some point I may have to block certain IP addresses or HTTP Origin referrers because of abuse.

I chose the "Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)" license to start the project without worrying about legal issues. Because the API is offered without API key restrictions, it could also be abused.

The legal way to remove the non-commercial limitation, is to "ask" if your company can use it.

DWD is quite unrestricted. Data is also processed, mixed with different models and many weather variables are calculated afterwards using other models. It is not showing raw data directly.

As far as I interpret terms correctly, I have to indicate that I am using open-data from DWD, but for value added services, this attribution is not required for all applications using it. I will do some additional research in this regard, whether applications using Open-Meteo should also display a disclaimer.

That's cool :) I like this idea.

Yes, please use the WMO weather code for that. I will also add precipitation probability in the next weeks. At >50% precipitation probability the umbrella hook light could also switch on.

Now I got it! Sorry, I did not get your point in the beginning. Compressed textures could be really interesting. So far I have no experience in this regard, but I will look into it

This is indeed a concern to me.

As others pointed out, blocking IP addresses is an option. Blocking access based on the HTTP Origin header is also possible.

I hope I can go on for some time without being to restrictive. With a fairly quick API that can scale with cheap VMs it should be feasible to keep it running at low expenses even at higher utilisation.

Thanks for the hint. I did not know them yet

Including ensemble forecasts into Open-Meteo is already on my backlog. And yes, this will return those 90% confidence intervals.

Forecast variability is difficult to express. For example precipitation probability describes how likely rain is given the forecasted atmospheric conditions. Instead some people assume it is raining in 10% of an area.

Making use of probabilities it also tricky. If you run an ice cream shop, with high operation cost and there is a 50% probability of rain, do you open with the chance of barely anyone coming or keep closed to save cost?

Ensemble forecasts also take up a lot of disk space, because each model ensemble is stored individually.

Thanks for pointing me towards ASTC HDR. I did not know this one. Compression will be an important part to build a longer historic database. I tested zstd, but it only had a ratio of 30%.

Currently I am using fixed size files, because I update all weather data "in-place". I can calculate which bytes to read for a given geographic location. If data is compressed, this would be very hard to do.

Currently only the past 2 days are available. To add more historical data is in the backlog [0]. Do you have an application in mind? Which weather variables over which timeframe?

[0] https://github.com/open-meteo/open-meteo/issues/27

Thanks for your offer. Unfortunately I cannot improve the weather forecast quality with a single station.

National weather services collect all kinds of measurements and observations from weather stations, air planes, balloons, satellites and radar to "assimilate" the global state of the atmosphere. Usually this is full of gaps, because only a fraction of the world is covered by measurement stations.

Once hour 0 is assimilated weather models start to calculate how different processes like solar radiation, clouds and winds change this state.

As Open-Meteo only uses the end-product of this chain, personal weather stations cannot be included anymore.

If you want, you could try to use machine learning and Open-Meteo weather forecasts to build your own tailored forecasts. Hopefully this idea will be picked up by someone with instructions on GitHub.

You are right! The coordinates in the graph have the wrong label. I will quickly change it. Thanks!

API / processing: Swift + Vapor with modules in C, a lot of libraries to work with scientific data formats. Hosted on cheap rented VMs with Nginx in the front.

Frontent: Bootstrap, jQuery

Gilab CI/CD pipelines with deploys

I do not actively prevent commercial use. I stated a non-commercial license as precaution to prevent all kinds of legal complications.

Mainly I also intent to reach open-source developers to make access to weather dat easier without the need to instruct users to sign-up for an API key somewhere.

The high-resolution NOAA GFS models will soon be integrated as well. Data are integrated seamlessly. The API will automatically select the best weather model for given coordinates. In the US the 2 km NOAA GFS model will then be used as default for the first available 60 hours.

• Soil moisture data is based on DWD ICON models with up to 2 km resolution in central Europe.

• ET is based on latent heat flux. This is not the potential evapotranspiration or ET0 reference evapotranspiration. I did not dive deep into the actual radiative transfer model for latent heat flux, but you can find it in the DWD ICON description.

Celsius is already the default temperature unit. You can optionally switch to Fahrenheit.

For wind speeds, all typical units are available as well. Let me know if I missed one.

Do you know an open-data source for ocean visibility data?

I want to integrate regular wave forecasts in the next month, but I am unaware of something that can improve dive visibility.

Yes, the API part is using Vapor. But there is not much code that actually relies on Vapor. It is mostly doing request parameter parsing and routing.

I use simple 2D lat lon grids for different regions and resolutions. E.g. the global grid uses 2879x1441 pixel grid. The third dimension is time. All data are stored directly as floating points on disk. Because I know the dimensions of the grid, I can read data exactly at the correct position on disk. I use Float16 as well, which saves me 50% disk space compared to Float32.

Fancy hexagons like H3 are not necessary. They could reduce storage by ~30%, but require much more effort and I have to "modify" the forecast. I keep forecast data from national weather services as untouched as possible.

Kachelmann seems to be more media oriented. I am not aware that there is a free API offer.

It looks like Kachelmann is also integrating weather forecasts from different national weather services and the paid version from ECMWF. I am sure the "HD" forecasts mainly use DWD ICON models as well. In this case, data quality would be the same.

I also spend some time to carefully integrate local and global weather models. For solar radiation forecasts, a clear sky radiation model is used to correctly interpolate data to 1h resolution. In the end it is a tradeoff between simple API with low operation cost and data quality/amount.

The host is netcup in Germany. I use regular KVMs.

Yes, the amount of storage can be an issue, but I want to stay below 500GB of hot-data. One bottleneck is network traffic to copy updated files after each weather model update.

My binary files are just plain Float16 files without an meta data. Logically they similar to 3D Zarr files. I know exactly which bytes I have to read and the kernel cache helps a look to keep it fast.

In theory this data does not have to be a file on SSD. I could also use a block storage directly or request data from S3 via range requests.

One hopelessly over-engineered approach would be to use an `AWS EBS io2 Block Express Volume` use `Multi-Attach` and spawn up to 16 API instances to serve data from it.

For now I only offer 2 days of past weather data.

Most likely I can store some basic weather variables like temperature, humidity, wind, precipitation and weather code for a couple of weeks. I will keep you request in my backlog: https://github.com/open-meteo/open-meteo/issues/27 Thanks!

My personal preference for Swift was for sure a major aspect. I also like to profile my code and see how much I can out of it. It is more an academical challenge to compare performance mmap vs file reads calls, but I enjoy it :)

With Python I always struggled to get performance to a very high level. I am sure it is possible, but Swift with some C code was more natural for me.

There is no funding. Costs are negligible. Currently it is running on a cheap rented VM. Per single VM, it should scale to 3+ million API calls per days easily.

To keep it simple, I also do not want to have any income with it. In case larger API consumers are on the horizon, I will contact them and ask to sponsor a couple more VMs.

The API backend is written in Swift with some code in C (conversion Float16, interpolations, some solar radiation libraries). Data are read directly via mmap from local NVMe SSD disks. To scale, I can add new VMs and copy my binary files.

There is no frontend cache and I want to keep it this way. Next steps include point-of-presence API virtual machines in different counties. Via GeoDNS this is even faster than a CDN.

Peak API usage was around 50.000 API calls per hour, but on average it is quite low.

I also tried brightsky.dev, but it is limited to weather stations in the MOSMIX system. IRRC only Germany. In terms of forecast quality, MOSMIX should be great!

One goal of mine, would be to provide historical data of the past months to enable users to run their own statistical forecasts combined with weather models. It is actually quite easy to use measurement data and correct forecasts using simple machine learning models like random forrest. If you compare wind speed measures and forecasts, there usually is a simple statistical error that can be reduced easily. Maybe in the next month I will to some tests and example code

At whatever time of day you call the API, data always starts at 0:00. Either UTC or the time zone you selected.

Many weather APIs follow weather model runs. Therefore in the evening, data suddenly starts at 12:00 UTC regardless of the timezone. It is quite a pain, if you want to build an app displaying todays data, but in the evening you do not get data from the morning anymore.

With Open-Meteo multiple weather model runs are merged together and partly overwritten. At around 4:00 UTC, the first weather model with data starting at 0:00 UTC arrived (usually called 0z run). 3 hours later, the 3z run arrives with data starting at 3:00 UTC. After a couple or model runs arrive, it is quite a mix out of different weather model runs. If done right, you notice nothing of this behaviour ;)

    < 0z run >
      < 3z run>
        < 6z run>
          < 9z run>

Data is constantly merged into a time-series. The API then selects the appropriate start time, which is set to 0:00 UTC (or if a timezone is set, local-time at 0:00, which could be 4:00 UTC depending on the utc offset).

Yes, unix-timestamps must be in UTC+0. If a timezone is set, data still starts at "2021-09-13T00:00", but this is now local-time. With a 4 hour UTC offset, I would have to set the unix-timestamp to "2021-09-12T20:00" and the developer has to correctly reapply the UTC offset again to get local time. This can be done, but is prone to errors.

Thanks for your suggestions!

I will add precipitation probability as soon as I include 14-day ensemble weather models. Deterministic whether models usually do not offer this directly.

I also really like the 2 days historical access :) At some point I would like to add more, but the storage requirements skyrocket quickly. Not only in space, but also disk IO to perform in-place updates.

For air quality data I want to add first all variables individually. AQI can be calculated afterwards on-the-fly. Some AQI indices require data from the previous hours, but it should work well with 2 days past data. For sure I will directly integrate the European and US AQI.

I considered unix-timestamp instead of ISO9601 string timestamps. Working with unix-timestamps is a bit trickier if you also want to use timezones. For the start, i settled with ISO8601 timestamps. I might consider adding unix timestamps as an `&timeformat=unix` parameter to the API. Regarding saving bytes in the output: I would like to implement protobuf as a format and also use it to build some SDKs. Especially with large floating point arrays, this saves a lot of parsing time and preserves precision.

All your suggestions are now kept in the Open-Meteo backlog https://github.com/open-meteo/open-meteo/issues

Creator from Open-Meteo here, I build a small, but very fast and precise weather forecast API for non-commercial use. I am a private individual working on it in my spare time.

Open-Meteo started as an exercise to process weather model data from the German weather service with up to 2 km resolution. Their forecasts are great, but hard to use for non-data-scientists who regularly work with NetCDF and GRIB-files. Using this data in simple apps, websites, your home-automation software, or robot lawn mower is complex.

The Open-Meteo API makes using this data easier. APIs accept standard WGS84 coordinates and return a weather forecast for 7 days in hourly resolution.

The forecast quality is surprisingly good. Open-Meteo includes global and regional weather models. Global models use 11 km resolution with up to 180 hours of forecast. Local models vary between 2 and 7 km resolution and 48 to 72 hours. Updates every 3 hours. The best model is automatically selected and combined to produce a single 7-day hourly forecast. Currently the best forecast model coverage is in Europe. Models for North America will be integrated next.

Under the hood, all data is stored in binary files using Float16 and updated in-place after a new weather model arrives. The API is very efficient. Returning weather forecasts takes usually less than 5 milliseconds. Internet latency is usually much higher.

All data is offered for non-commercial use. With speedy APIs, all data can be served by just a couple of virtual machines for less than a coffee a day.

What’s next? Some important features are still missing like daily aggregations, additional weather models, ocean, and air quality forecasts. Additionally, I would like to deploy some servers in North America and Asia to improve latency.

The project went live 2 weeks ago and is slowly being used. I would be grateful for feedback, suggestions, ideas, and questions.

All documentation can be found at https://open-meteo.com/en/docs