Hacker News Reader

Comment by lacker | Hacker News Reader

It seems quite strange if computing power is really the bottleneck here. What sort of architecture would you have for a weather prediction system where you would spend more on computing power than researchers' salaries?

View on HN

There is a very good reason computing power is the bottleneck and it is because of how the weather has to be modeled. Weather models simulate the conditions in the cells of a lattice that is overlaid over geographic areas. To improve accuracy, the cells need to be shrunk to smaller sizes as they have been for a long time. However, every time the size of the cells in the lattices are halved you need 8 times the computing power to perform the calculations of the 3D space. Because of Moore's Law, it takes many years for it to make sense for weather centers to upgrade their infrastructure and see significant improvements in their predictions.

Cells also dynamically influence neighboring cells in all dimensions, making the highest performance, lowest latency interconnects essential. Interconnect pricing is a significant chunk of these large system costs.

I occasionally hear statements wondering why weather modeling can't simply be distributed to desktop or cloud and can't, for some strange reason, get people to understand that connectivity is almost as important as computation itself for this.

We have a few clusters at work that we use for numerical weather forecasting using the WRF weather model and some other custom applications. And, indeed, the InfiniBand interconnect was a hefty portion of the overall cost of the clusters. At one point there were people considering Gigabit Ethernet instead of IB, but thankfully it was shot down. The lower latency with IB has a greater impact on MPI-based applications than some people realize.

So, really, no commercial "cloud" product meets these needs? So many other industries are beyond managing hardware; seems like an obvious opportunity for a product.

Azure supports RDMA over IB on certain node types but (my 2 cents) without more detailed information & control over topology it's hard to run non-trivial MPI jobs in anything close to an optimal way. The hiding of the "secret sauce" that cloud vendors do may not be important for most commercial users but it's a roadblock for a lot of HPC.

At that point you just call it a supercomputer. Private companies already buy time on government/academic supercomputers, so I don't know if there's a compelling opportunity there since your startup costs would be immense.

AFAIK, weather models use a mesh which is much coarser in z-direction. So it's a factor of 4, not 8. Another issue is that increased resolution isn't much helpful unless you also have higher resolution initial data (measurements). Finally, handling uncertainty means you have to do multiple runs. I believe the wall clock runtime of a single weather simulation has been stable at about 30 mins. for decades now, and increases in computing power is used to either refine the grid or to run more simulations with slightly different initial data. Then there's also the problem of postprocessing and analysing the huge amounts of data you end up producing.

Actually, you may want to cut the time step as well, so it could be a 16x computational requirement.

Right now, we spend far more on satellite data than we do on salaries and computers.

Reply on news.ycombinator.com