Comment by Afforess | Hacker News Reader

Comment by Afforess | Hacker News Reader

Parent Full thread

The Euro model win only demonstrates the effects of budget cutting on results. Meteorology computer modeling is nearly entirely dependent on supercomputer time, and the time and quality of supercomputing resources is a direct function of money available. Meteorologists can discover better equations for modeling the atmosphere, but without the computer resources to implement these models, they are useless.

Weather models are an O(n^4) problem (you solve in x, y, z, and time), so the computing resources needed to model the globe are vast. Right now, the GFS (Global Forecast System) resolution is 18 miles. That means it has to treat the globe as a grid of 18 mile cubes. 18 miles is big; it allows for a lot of smaller-scale weather features to slip through the cracks.

Ultimately, Less money -> Less computer time -> Worse model resolution -> Worse model output -> Worse forecast. Meteorology is one of those rare places where throwing money at the problem does fix the issue.

https://www.ncdc.noaa.gov/data-access/model-data/model-datas...

There is still a big gap of live atmosphere observations that are being fed into these models. Even if you throw money at them, you will not end up with increasingly better results since the initial state of the model will be the same. Throwing money at the model runs and the supercomputers will cease to provide positive returns at some point (and I think the Euro is probably near that point).

We need a dramatic increase in the number of observing stations as well as funding increase for the supercomputer time. I'm personally very interested in what we can do with smartphones right now; there are billions of internet-connected barometers that the Euro and the GFS do not use. Project like PressureNet[1] and Sunshine[2] are making huge strides there, and I think the future is very promising.

[1]: Android: https://play.google.com/store/apps/details?id=ca.cumulonimbu...

[2]: iPhone: https://thesunshine.co

You're right, but throwing money at more observation stations, satellites, and balloons would work as well. The general "throw money" solution would work.

> That means it has to treat the globe as a grid of 18 mile cubes

Although this is made somewhat easier in that the vertical component is done in pressure "levels", somewhat simplifying the equations to be solved. However that shouldn't underestimate the scale of the problem. IIRC a model with a surface resolution of 100km means solving for something of the order of 30 million variables for each time step.

Full scale climate models can get even harder, as you need to couple an ocean circulation model to a atmospheric circulation model - ocean circulation is less important for a weather model as its time scale is of the order of up to 1000 years.

It's not just inadequate computing power. Cliff Mass has a good run-down on his blog: http://cliffmass.blogspot.com/2012/03/us-fallen-behind-in-nu...

The number two reason on Cliff's list is "The U.S. has used inferior data assimilation", but that will be changing next week! The Ensemble Kalman Filter goes live next week: http://www.nws.noaa.gov/os/notification/tin15-43gefs.htm

It's unlikely this will cause an upset (the US is far behind), but it should at least close the gap.

Very insightful comment. I have long been interested in what the actual approach and factors are to generating better forecasts and you explained the important parts. Thanks!

If you have access to Nature, you might like this article [1]. Even if you don't, check out Fig. 1 to see how forecasting has been improving.

[1] "The quiet revolution of numerical weather prediction", http://www.nature.com/nature/journal/v525/n7567/full/nature1...

Also at http://www.uvm.edu/~cdanfort/nolink/atmos/nature-review-2015...

Thanks for both of these (also sub comment). These were great reads, kept me up last night.

Why give money to a solution before exploring other options? Decentralization, for example, seems like a more natural way of distributing meterological forecasting workload across the planet, by colocating sensors and computation on clusters of devices in close proximity. Decentralized computing is a natural fit for weather problems.

The mathematics that needs to be solved to produce weather forecasts isn't embarrassingly parallel in the way something like Ray tracing is or in the way the BOINC projects like Folding@Home take advantage of. The mathematics involved is attempting to model the continually interacting, global state of weather across the planet in order to determine what it will be like at various places on the planet.

These models are extremely reliant on being able to "share" data between parallel operations, if computer core 1 is doing a grid cell, and computer core 2 is calculating things about a neighbouring grid cell, it will need to know things about the grid cell core 1 is working on. These models run on supercomputers in order to facilitate efficient parallel processing in an environment where each node can access the memory of surrounding nodes since promptly accessing such memory is necessary for the models to be computed "faster than real time".

Distributed computing "across the planet" only works with low-bandwidth/low-latency tasks. Take a look at the various distributed internet projects (the Great Internet Mersenne Prime Search, Folding@Home, etc) and they are ones that process for a long time and send back a small amount of data, typically to a single server.

People use supercomputers for problems, like weather forecasting, which depend on high-bandwidth, low latency interconnects. For example, the Cray CS-Storm (which the Swiss National Supercomputing Centre will be using) supports "QDR or FDR InfiniBand with Mellanox ConnectX®-3/Connect-IB, or Intel True Scale host channel adapters" (quoting http://www.cray.com/sites/default/files/resources/CrayCS-Sto... ).

To prevent congestion, the supercomputer network topology might even be wired so each pair of neighboring nodes has a dedicated connection.

Err, "with low-bandwidth/high-latency tasks".

Reply on news.ycombinator.com