Met Office forecasters set for 'billion pound' supercomputer | Hacker News Reader

Met Office forecasters set for 'billion pound' supercomputer

bbc.co.uk

36 pointsaluket35 comments

I guess this is not unrelated to the ECMWF departing Reading for Bologna https://www.ecmwf.int/en/learning/workshops/ecmwf-bologna-20...

This is unrelated. The Met Office is going out for a new platform to run daily forecasts rather than climate models, as the current machine reaches end of life in 2022.

I stand corrected

One interesting fact that I learned from Nate Silver's book, The Signal and the Noise, is that weather forecasting is a four-dimensional problem (space + time), so to produce a forecast that is twice as detailed requires approximately 16x the computing resources. Historically, the resolution of weather forecasts has doubled roughly every eight years, in line with with Moore's Law.

Is there a similar rule of thumb for how accuracy tracks with the added resolution?

Do you mean added resolution of input data, or improved resolution at which the models are run? Input data is frankly very limited - IMO we are barely measuring the atmosphere. Model runs are higher resolutions are only good if there is sufficient input data to seed the models.

It's my lifelong passion to increase the usable input data (live atmosphere measurements) that models can assimilate. My latest attempt is to use the barometers in phones to create billions of new 'virtual weather stations'. My US-only Android app (iOS and international coming soon) is

https://play.google.com/store/apps/details?id=com.allclearwe...

You can see an animation of the data recorded in the Orlando, FL area when Dorian was churning off the coast: https://www.allclearweather.com/hurricane-dorian

The data requires significant QA and bias correction to use, but it is possible (see Cliff Mass research papers) and the trends in the data are already clear and usable, regardless of actual pressure value recorded.

Let's add a source code link for Android background/foreground sensor access that I wrote to further these goals: https://github.com/JacobSheehy/AllClearSensorLibrary

I heard that it's twelve dimensional. Used to work with a guy who's PHD thesis was on the diffeq of weather predictions.

'if twelve cows are lying down it's gonna rain'

QED

The time-spatial discretization is 4D. There are many different state variables within each grid cell.

Vorticity patterns are often treated as an additional set of 3 dimensions, because they require continuous differentiation. Modern weather forecasting software is a beast.

I've done some reading through the literature on the dynamical cores of weather models. It isn't really true that vorticity is modeled as an additional set of dimensions.

Vorticity and divergence are an alternative description of the fluid velocity. They are the curl and div of the fluid velocity, respectively.

Just as the fluid velocity may be discretized in 3 spatial and one time dimension, the fluid's vorticity and divergence may be discretized in three spatial and one time dimension.

so 32 years ago they could forecast an hour into the future?

It's more like the ranges of uncertainty for given time horizons were much larger 32 years ago then they are now. Take a look at https://www.globalweatherclimatecenter.com/tropical-cyclone-... for hurricanes in particular: the 5-day forecast is now more accurate than the 2-day forecast was in 1990.

Ah - I momentarily forgot that the Met Office is in a country where “pound” is a measurement of money, not weight.

This made me curious. Apparently supercomputers can weigh 1 million pounds [0]. So a billion pound supercomputer in the US would be ~1000x more powerful than a billion pound supercomputer in the UK and cost a few percent of GDP to build.

In some countries, one "billion" means 1,000,000,000,000 instead of 1,000,000,000, so there it would be exactly 1000x more powerful than in the UK.

Wasn't that originally a British thing? But we adopted the American billion ages ago. Are there other countries where it's still used?

germany, holland, belgium, maybe more

Seems there is not much information yet on the actual hardware. Quick search found this:

https://siliconangle.com/2020/02/17/hpes-cray-tapped-build-m...

Eventually reaching 145 PFlops

The Met Office didn’t share further hardware details other than the fact that the supercomputer will incorporate graphics processing cards.

There are also some voices that attribute bad local weather forecasts to closed weatherstations and errorprone digital replacements to manual measurements... But hey, new supercomputers are cool!

The Omega Tau podcast did an interesting episode about weather modeling at the European Center for Medium Range Weather Forecasts

https://omegataupodcast.net/326-weather-forecasting-at-the-e...

Interesting they’re talking about colocating it in EEA countries. I see the rationale for Iceland and Norway, but why specify them as EEA? Is there a post-Brexit strategic angle to this? (considering the large sum of public money involved)

They are not specified as EEA countries. The EEA is specified as the constraint for the supercomputer's location - Iceland and Norway are given as two possibilities, but after that constraint.

How would you phrase that part of the article if indeed any EEA country could be the location of the new computer, and you wanted to include Iceland and Norway as two candidate countries?

If the goal of the remote location is access to stable renewable energy wouldn't a South European location make more sense ?

I haven’t researched this, but I imagine the reasoning goes something like this: Iceland has geothermal, and Norway has hydro. Both will need less storage than solar (perhaps none?).

Other more political thoughts are:

Southern Europe looks like Portugal, Spain, Italy, and Greece. Perhaps Iceland and Norway seem more politically stable?

And maybe this will mend some fences with Iceland after the UK seized it’s banks assets under the terrorism act (although that’s going back a little bit now).

Edit: Typo

Spain, and much of southern Europe, has good solar, but not that much renewable at night I think. The setup for wind is better in slightly more northern europe. Iceland has geothermal that never fails, and Norway has hydro.

The article says "easy sources of clean energy" : that doesn't actually fully equate to "renewable energy".

Assuming that "clean" really means "low carbon", then only majority nuclear/hydro/geothermal electricity grids can currently achieve that. Wind/solar on the other hand are intermittent, and always need to be complemented with "dispatchable" energy sources to handle the base load.

That can either be hydro/geothermal if you were blessed with the right geography (like Iceland or Sweden), nuclear if you weren't but are pragmatic about it (like France), or coal/gas if you got scared of nuclear but still have a large country to power (like Germany).

I'm stressing the latter because, even as Germany is rightfully praised as a renewables champion that invested billions to be 70% wind/solar powered on a very good day, that's all in vain when it comes to climate change : coal/gas is so bad that their average carbon intensity of electricity production is still mediocre (see http://electricitymap.org/)

So, renewables doesn't always mean low carbon. If that's the primary concern for the location, France is probably their best bet (nearly as low carbon intensity as Iceland, and much closer to the UK)

> Wind/solar on the other hand are intermittent, and always need to be complemented with "dispatchable" energy sources to handle the base load.

I've seen people claim here that battery storage already represents a good solution to that problem. Elon Musk's battery storage project in Australia seems to be successful and powering a supercomputer would probably require a much smaller installation.

Latency would be too high outside of Europe

I'm surprised to see a supercomputer cross the 1 billion pound/euro/dollar mark.

Previous recent supercomputers seem to have cost in the low nine figures.

I realize the price tag includes a decade of operation but that still seems like quite a leap.

The "cost of ownership" is often approximately 33%/33%/33% for capital cost (annually), support (annually) and users& operations. Of course being government they probably don't account for the costs on an accrual basis.

Two machines, 5 years apart, and 66% non-hardware for ten years is, what, 250 or 300 millions for the pair of them?

I'm not sure what exactly they're including in the operating costs. If they include salaries for researchers, meteorologists, programmers, etc. I could definitely see it hit a billion over 10 years. But then it would seem a bit misleading to call it a "billion pound" supercomputer.

A previous Met Office supercomputer purchase was discussed here: https://news.ycombinator.com/item?id=8519820

From an outside perspective I have to wonder if a billion pound investment in forecasting/science would not deliver better long term ROI.

Would it do so in just a few years though?

The pragmatic approach is to invest in both tools and research.

It depends– if you're trying to run a specific algorithm on your new supercomputer, then you'd almost certainly be better off paying for researchers to optimize or improve on that algorithm. If that's the situation (which it is for e.g. weather forecasting or computational fluid dynamics), then a billion pound supercomputer is likely to be more of a boondoggle than a sharp-eyed investment. A good implementation on a desktop can beat a bad one running on a supercomputer.

But if it's a time-sharing system, then it might not matter as much. The supercomputer at my university tends to run a lot of one-off jobs like an experiment repeated thousands of times with different parameters. On a desktop that might take weeks, but if run in parallel it's like a couple hours. Tightly optimized code might bring that down to an hour on the cluster (or a mere week on my home PC) but I wouldn't bother because making the code more efficient might itself take a week or more. So the fastest way to get the results I need would be to just run it on the supercomputer.

Reply on news.ycombinator.com