Show HN: Ranking weather models by how their forecasts turned out | Hacker News Reader

Show HN: Ranking weather models by how their forecasts turned out

nickleenders.github.io

7 pointsnickoteeno6 comments

Weather apps all claim to be accurate but never show their work. So I built a scoreboard that checks: it takes the forecasts each model published (ECMWF, GFS, ICON, AIFS and others, plus Apple Weather, Foreca, OpenWeather and Visual Crossing), waits for the weather to happen, and scores temperature, wind and rain against observations.

A few things that surprised me: - AIFS performs very well, yet almost no commercial apps give you access to it or uses it in their blend (I suspect some do without disclosing it tho) - Foreca scores surprisingly well compared to other apps and raw models - ICON is very accurate around the mediterranean, but performs quite poor everywhere else

There's also a history page that scores each model back through its full archive (about 5 years for GFS) to see if forecasts have actually gotten better.

It's a static page and open-source: https://github.com/NickLeenders/verisky-scoreboard. Public models are scored in your browser against Open-Meteo's archive. Commercial scores come as small aggregates from my server, because those providers' terms don't allow redistributing raw forecasts.

It powers an app that does the same per location in more detail, link is on the page.

Happy to answer questions about the scoring method.

Open articleView on HN

So this doesn't need to be hourly. Use the ECMWF IFS 00 and 12 as observations only.

RMSE really favors average models, AI models are bad... but look good in statistics because they are so average. Look at the output of AI models visually on a map and they are just a blurry mess.

In my area AIFS and AIGFS get the top results but the actual best performing model (because I'm a weather nerd) ends up in third place.

There also seems to be some issue with short and long run models in you comparison. I also see a few models on the list that don't cover my area so it looks like you are not querying for just the specific models but the model+ECMWF fill.

Fair point, what would you recommend instead of RSME (for temp and wind as rain uses F1 anyways)? I query the seamless variant yes, which for longer lead times replaces with ECMWF IFS. I'll take a good look at that. Which would you normally say is the best for your area?

Really interesting!

Might be useful to expand the full name of each model somewhere (perhaps with a one para description and/or or a link).

On the trends tab, would it be worth zooming in a little? There's less value in showing 2021-24 since only GFS data is included for those years.

Adding a link would be fair indeed! On the zooming in, you're right, I thought I did that, and I did for all but temperature, which is default, I'll change that right away

very cool project. I imagine there is a commercial market with the weather polymarket betting. I would love to see a legend, or some guidance on comprehended your data.

I will be self promotional here: my project, slickfast, is free and open source. SlickFast is all about visualizing data, its super powerful and made for agentic workflows. It could buttress your project and won't cost anything to try it out, or free if you meet AGPL. I'd love to talk to you about it.

Thanks! I haven't heard about weather polymarket betting, sounds fun. I will have a look at slickfast, do I understand correctly it's an mcp?

Reply on news.ycombinator.com