NOAA upgrades the U.S. global weather forecast model
noaa.gov
noaa.gov
P.S.: Anyone notice that monitoring "Climate" was absent in the government announcement?
[0] https://www.amazon.com/dp/1324002646
[1] https://en.wikipedia.org/wiki/Michael_Lewis
[2] https://oceanleadership.org/trump-taps-accuweather-ceo-head-...
But I dug around for some information to maybe get you started.
Installation: https://madis.ncep.noaa.gov/doc/INSTALL.unix
API: https://madis.ncep.noaa.gov/madis_api.shtml
Data restrictions: https://madis.ncep.noaa.gov/madis_restrictions.shtml
Another resource that may help: https://press3.mcs.anl.gov/forest/regional-models/global-dat...
When I was working on this stuff, I found that a DFS on various government subdomains (like MADIS) was the best way to find information. It was tedious, but it worked.
It's also helpful to put on your fortran hat. For example, I once attended a Haskell meetup where someone wrote a parser to deal with parsing binary files from NOAA. I also was in a meeting (with some NOAA folks) once where I was asked if I "would prefer an ASCII file, or a binary one". This is not a world that operates on JSON or XML. Expect binary blobs with flags (bits) that change the meaning of other flags in fun and exotic ways. The binary nature of the data can help with data throughput limits, but boy is it a pain to deal with.
That bring back memories... As a government contractor I've had to work with sensor data (seismic, radar, etc) in various formats that were developed well before the rise of XML and JSON :(
My favorite was a mixed ASCII and binary format, where each data record in a file had an ASCII header that described the format of the following block of binary, and pretty much anything could be different between records, even within the same data file (Time units? Integers? Floats? 16 bit integers? 64 bit? Big/Little Endian?).
I had to write a parser for that :'(
When I started the project, I looked online to see if anyone had done any previous work on this thing. A vendor was selling a GUI for the thing for $2000, I scoffed at the price and started working it myself. By the time I was done, it had probably cost my employer more than that but at least we had our own code that could connect to whatever you wanted rather than a GUI with a no API.
MADIS is a bit dated, data dissemination that uses OPeNDAP [1] and ERDDAP [2] is much friendlier.
[0] https://github.com/Unidata/siphon
[1] https://opendap.github.io/documentation/
[2] https://coastwatch.pfeg.noaa.gov/erddap/information.html
https://www.zdnet.com/article/europes-big-weather-supercompu...
As far as I know other than this model upgrade, there are no major investment being made in American weather forecasting.
All of this for same day forecasts, not even 2 days in advance.
This is really handy as national models can be much better for short-term and higher resolution predictions.
It has the widest range of models freely available that I'm aware of, including the commercial ECMWF model.
Globally, we have GFS (FV3) and GDPS in the free version, and DWD ICON in pro.
For north America, we have NAM 12km, NAM 3km, HRRR, RDPS, HRDPS in Pro
For Europe, we have DWD ICON EU, DWD COSMO-D2, ARPEGE, AROME and HIRLAM in Pro.
If you double-tapping of a graph, you can compare the forecasts quite easily.
Sorry there is no iOS or web version, nor ECMWF (it's too costly).
[0] https://flowx.io
The forums really light up for big storms with lots of discussion and insight.
https://cliffmass.blogspot.com/2014/11/the-other-radar-gap-e...
https://www.nssl.noaa.gov/tools/radar/mpar/
Alternatively, Roberts Field appears to be a major commercial air hub in central Oregon. You might argue from a safety perspective to your Congressional representatives (perhaps in concert with local air carriers and AOPA) that the airport needs a TDWR station (cost will be ~$4MM-8MM), which could also provide NOAA with the necessary weather surveillance data. Thunderstorms aren’t common on the West Coast though, hence the lack of TDWR stations in West Coast states. If you pursue this route, you'd want to get funds for this into some sort of federal transportation bill, as part of enhancing the safety of the air transportation system.
https://en.wikipedia.org/wiki/Terminal_Doppler_Weather_Radar
https://www.smhi.se/en/services/professional-services/microw...
Edit: Downvotes for simply asking a question. sigh.
I also recall the ECMWF had surprisingly accurate long range forecasts based on ensembles. It could predict 500mb heights out two weeks, no sweat.
Re: your comments... My guess is that a gpu isn't suited for use in an operational model due to data access patterns (and possibly not even helpful with the solver). But again, I'm not a computational pde guy. Also, perhaps machine learning would be useful but that would be post-processing or perhaps parameterizing sub-grid phenomenon. There's already a process called model output statistics (MOS) for adjusting raw fields from a weather model.
I've been out of the field for ten years now, but it's really nice to see improvements to the core physics to this degree.
I'm still skeptical of your supposed two week 500 heights forecast from the ECMWF model. I live near the western Pacific (i.e. the data hole) and it's really easy to find crazy model solutions after 7 days. And I'm pretty sure you weren't looking at the Southern Hemisphere.
You're probably right to be skeptical, for the record I was only a forecaster for a short period of time over ten years ago... didn't even serve my full four year commitment as I volunteered to get out under the Air Force "force shaping" at the time. I was stationed near Rammstein and we created forecasts for Europe. I was referring to the ECMWF ensemble products, specifically.
Where GPUs shine for PDEs is if you have a lot of extra work for each node, for instance if you have complex chemical reactions or thermodynamics, or if you have a high-order method that requires lots of intermediate computations.
If you don't believe me, you can download the PETSc code and test the ViennaCL solvers versus the regular ones.
> A modern 5-day forecast is as accurate as a 1-day forecast was in 1980, and useful forecasts now reach 9 to 10 days into the future (1). Predictions have improved for a wide range of hazardous weather conditions, including hurricanes, blizzards, flash floods, hail, and tornadoes, with skill emerging in predictions of seasonal conditions.
> ... Data from the NOAA National Hurricane Center (NHC) (13) show that forecast errors for tropical storms and hurricanes in the Atlantic basin have fallen rapidly in recent decades.
This covers some decades, to be sure, but it's a pretty big improvement.
In [2] there is a slightly different claim, "A modern 5-day forecast is as accurate as a 1-day forecast was in 1980, and useful forecasts now reach 9 to 10 days into the future."
Chart 3.2 of [3] shows this; by 2001 the 5th day forecast improved to be as good as the 3rd day of 1980, establishing the trend line.
Googling about yields a few other studies and articles in a similar vein.
It is important to note that forecast improvement is not linear in effort. It takes more complete and accurate sensor data and far more computation to extend the forecast on the out days due to the chaotic nature of the mechanisms modeled.
[1] https://rmets.onlinelibrary.wiley.com/doi/full/10.1002/qj.25...
edit: I see that neuronexmachina found the same article. It's a good read if you want an overview of how weather prediction has changed.
On the other hand, I suspect it might not be noticeable if the forecast you always read just says "40% chance rain, high 80, low 50". It might be more noticable if you look at the hour-by-hour forecast for a specific location and see when the rain is predicted to start and end.
But by mentioning machine learning, I'm guessing you are looking at a different timescale, i. e. "within the last two years". And any progress in the short term will be slow compared to what we have seen in other domains such as image recognition etc.
I'm no expert on weather forecasting, but I believe the explanation may be that forecasts have long been (among the) best financed "big data" problems out there. That means they incorporate lots and lots of domain-specific work. As a result, naive machine learning models currently still lag all the specialised work, which in turn isn't structured in a way to easily take advantage of progress in, say, GPUs.
Note these error statistics do not represent true model error as the official track and intensity forecast -- while informed by model output -- are determined by human forecasters.
https://www.wrh.noaa.gov/images/mtr/sjc_pcpn_prob.gif
Since then, I spent hours trying to find this page again, unsuccessfully. All I have is this URL for San Jose. (Replacing "sjc" with other airport codes doesn't always work, since "mtr" is a region code?)
Anyone know where this precipitation data lives?
Missing Controller
Error: Climate.DashboardController could not be found.Was it accessing/presenting raw NOAA data? Or a different source? I notice the data is thru 1990; different from NOAA.
Is the php source available for reverse engineering?
With some minor effort you could extract the raw data, in JSON form, from https://www.wmo.int/cpdb/climate/climate_normal/per_country/...
If I remember correctly the data came from a CD-ROM with historical data. When we put it online the data was already 20 to 25 years old. It was nevertheless the most recent data that we had available, I don't remember the reason why we didn't have anything more recent. The PHP source is (or was) a CakePHP application, and honestly isn't that interesting. There was not more data in the PHP application than what is presented here.
EDIT: It was already a messy application when I got there, I cleaned it up as well as I could, but after I left it seems to have gone downhill again. Ah well. Not my problem anymore.
> The model paid close attention to conserving energy, mass and momentum in the atmosphere in each box. This precision resulted in dramatic improvements in the accuracy and realism of the atmospheric chemistry.
Global Forecast System > Future https://en.wikipedia.org/wiki/Global_Forecast_System#Future
Using AI and ML to make predictions about weather will likely not account for the conservation principles and might lead to ridiculous results (in some sense). Creating an accurate AI/ML model of a complex and chaotic system might lead to wrong predictions under extreme circumstances (e.g. predicting the weather >5 days out for an extreme hurricane) or under conditions where some implicit assumption has changed. One can at-least attempt to grapple with these issues when using finite volume. Under AI/ML you just have to hope your model is properly trained.
> The retiring version of the model will no longer be used in operations but will continue to run in parallel through September 2019 to provide model users with data access and additional time to compare performance.
Machine learning is all about finding an unknown function that underlies known data. This is sort of the opposite issue: we know the underlying function but can’t compute it.
That said, ML models are being applied to situations where the whether data doesn’t translate directly into known physical quantities, like satellite images (see https://developmentseed.org/projects/hurricane-intensity/)
So even if we had a forecast engine that would perfectly simulate everything given some start state, we wouldn't have enough input data to have an accurate start state.