Supercomputer quietly puts U.S. weather resources back on top | Hacker News Reader

Supercomputer quietly puts U.S. weather resources back on top

usatoday.com

113 pointsjonbaer77 comments

It's a very exciting time in weather forecasting. We're getting all kinds of new observations available, from cheap LEO satellites to smartphone sensors to smart umbrellas and everything else. And faster computers! And more parallel computers! Think about every single iPhone and every single Galaxy S in the world as a useful sensor array _and_ a parallel processor...

Weather forecasting accuracy is still constrained by both the available observations and the ability to run the forecast models. This new supercomputer won't be nearly enough to process all the new data that will arrive in the next few years, so the pendulum will swing back to the bottleneck being computing power soon enough. I think the most exciting things will happen over the next 5-10 years, as we end up with another 2-3 order of magnitude increase in the available observations to the models and we'll build new supercomputers to process them. I bet we can get incredible improvements, things like reliable 2-week weather forecasts.

No one in the foreseeable future is going to use cellphones as parallel processors. First of all, the communication latencies are way too large to coordinate anything usefully. Second of all, users are already upset about the lack of battery life available for their phones.

It's not exactly parallel processing, but the Dark Sky app already does its forecasting calculations on the iPhone: http://jackadam.github.io/2011/how-dark-sky-works/

> The reason we encode velocity data as an image is so we can pass it off to the GPU on the iPhone and iPad. Both the storm prediction and the smooth animations are calculated on the device itself, rather than the server, and all the magic happens directly on the GPU.

In case anyone starts wondering, in the article it says that they're assuming that atmospheric motion is linear, which they justify (reasonably) because they're only looking over short time scales. So, although it is a forecast, it's a pretty simple one ;-).

> Think about every single iPhone and every single Galaxy S in the world as a useful sensor array _and_ a parallel processor...

Sadly, until we have a breakthrough in battery technology, all of those will be parallel processors programmed to be asleep as often as they can.

Sure, that's true - but they can just as happily run their GPUs if they are plugged in and charging without any battery issues. Although for the moment you are right, they aren't that useful as a powerful distributed supercomputer. They ARE, however, really really good at collecting live weather data and sending that to the supercomputer to process. One step at a time.

Smart phones are good at collecting live barometric pressure data but not much else. Temperature sensors aren't very useful because they are poorly calibrated, you can't reliably tell whether someone is outdoors, and the reading can be thrown off by the heat generated by the device or the person holding it. UV sensors are seldom pointed in the right direction. And I haven't seen a phone with a wind speed or precipitation sensor yet.

I genuinely don't mean to sound passive aggressive when I ask, do you know anything about this subject or are you just guessing?

I don't know anything about it, but I can think of numerous ways to mitigate the issues you mention. E.g.:

* Does it actually matter that the absolute value of temperature sensors is inaccurate, or is it enough that they have good relative accuracy (do they?)

* Do you really need to tell if someone is indoors or outdoors given a large enough sample of users in a given area? That also seems like something you could detect with heuristics (work hours, jumps up/down in temperature).

There's a lot of ways to tease useful signals from noisy data. Is it really the case that smart phone temperature sensors in the aggregate are useless as an input into weather forecasting models?

I'm a graduate student in Environmental Fluid Dynamics and I deal with a lot of environmental sensors. The temperature sensors on phones are pretty accurate, but they're quite sensitive to small changes. This means that nradov's original statement is correct: Body heat and cell phone usage are absolutely going to affect the temperature sensor. For example, Sensirion makes a high quality temperature/relative humidity sensor and they recommend the PCB have cutouts around the sensor to offer proper ventilation and minimize heat transfer from surrounding components [1]. On the iPhone 6s, the pressure/temperature sensor is on an edge, but it's also right next to the NAND flash chip, so I'm not quite sure what the heating effects would be.

I can't answer much to your second question but I imagine it's easier in colder environments, especially if the phone is not in a pocket. In addition, here's a publication that claims to be able to get useful signals from noisy data, so I don't know if I would say the measurements are useless [2].

[1] Fig. 10 in https://www.sensirion.com/fileadmin/user_upload/customers/se...

[2] http://onlinelibrary.wiley.com/doi/10.1002/grl.50786/full

I'm not sure how useful the usual techniques for normalising sensors would be for moving sensors.

Fixed weather stations have corrective mappings associated with them. For example the station at the airport might be assumed to be 1 degree warmer than the generalised temperature for your city. It might be known to collect 5% more rain when the wind is northerly, and 8% less for southerly. These mappings are very important, because they let you shut-down a weather station, and build a new one in a different location, and compare the data from the two.

if the weather stations aren't in consistent locations, with discoverable local behaviors, then converting the raw data into generalised data about the area is going to be very difficult.

I also know nothing about the limitations of the temperature sensors on smartphones (I don't even know of any phones that have temperature sensors).

What I do know is that whilst teasing signals out of noisy data is achievable when the data are unbiased, the possibility of biases makes it significantly harder, especially when you don't know what the biases might be. This can be a significant problem with all sorts of data that you might think were fairly good, including both in-situ and satellite measurements.

I'm not an expert but have played around with sensors on some Android devices. The absolute thermometer accuracy will vary by device. When the temperature rises by 1℃ over the course of an hour is that because it actually got warmer, or is because the user moved inside or put it in a warm jacket pocket or ran another app that caused the CPU to heat up? In theory I suppose if you had thousands of devices in a small area uploading simultaneous temperature readings you could perform a regression analysis and extract a little bit of signal from the noise. But in practice I just don't see how you could ever get enough device density in a small area to make that work. Remember that in urban areas the vast majority of smart phones spend most of their time inside climate-controlled vehicles and buildings, so any temperature readings from those are worthless for weather prediction. But they do still give useful barometric pressure data because the pressure is fairly consistent inside and outside for a given location and altitude.

just curious, do you have references to smartphone sensors being utilized in numerical weather prediction?

i'd been browsing through some WMO reports and other associated NWP lit recently and i'd seen stuff on GPS-RO, but nothing on anyone assimilating "smart" devices.

Sure. I've been working on this problem for about 5 years now, starting with the collection of barometric pressure from Android devices. I'm currently working on this with iPhones at Sunshine [1], where we collect pressure data (along with other metrics), but pressure is the most valuable.

There are researchers who use this data - we have collected about 4 billion atmospheric pressure measurements that we have distributed for academic and government research. The primary researchers are Cliff Mass and his lab at the University of Washington. There are also groups in Canada and the US that are using the data. IBM is now also collecting and using smartphone pressure data through their mobile apps. [2]

Generally speaking, the current trend is to take the live data stream, run it through a quality-control algorithm and then use kalman filters in the WRF data assimilation package.

There are some papers published, but it is still early. I will find some links to papers if you'd like to read them. [3]

[1] https://thesunshine.co/

[2] http://www.nytimes.com/2015/10/29/technology/ibm-to-acquire-...

[3] Utility of Dense Pressure Observations for Improving Mesoscale Analyses and Forecasts: http://www.atmos.washington.edu/~hakim/papers/madaus_hakim_m...

ah, thanks, that's fascinating.

"It also learns every time you actively report to the community on sky conditions and hazards, translating this information into weather predictions."

does sunshine actively run a numerical weather prediction model like WRF?

it'd be great to assimilate all these extra sensors into the NWP centre's models, but i imagine things like cal/val and WMO agreements to share data might make things difficult for commercial companies?

Yes, Sunshine is running WRF!

And yes it would be great to have all these sensor readings available to NOAA, Environment Canada, ECMWF, and everywhere. I have made lots of progress in getting them to talk about it, but it's a long road before any government starts using this data in its own NWP models.

you're probably already aware of this, but there is some US legislation trying to open up commercial data to gov't entities:

https://www.congress.gov/bill/114th-congress/house-bill/1561...

there's some stuff being written this year as well for the side that i work (space). with all the upcoming sensor gaps, they're looking at alternative ways to cover.

This is really interesting stuff. How are you guys managing that scale of streaming data? Some kind of Kafka based stack?

If you'd like to submit your own weather observations from anywhere on the globe, the National Severe Storms Laboratory has created an app[0] to report them.

[0] http://mping.nssl.noaa.gov

I don't know much about it, but I know some people doing a startup all about these sorts of things

https://senseai.io

In other words. This is how the actual IoT revolution (if one can call it that) looks like.

I'm bothered by how the article says "three years ago European models delivered a blow to the U.S. weather apparatus". Better models by Europeans do not make the models by Americans any worse. Improved models help everyone.

That was the first thing I noticed, too. The fact that the Euro computers got Hurricane Sandy right was NOT a bad thing - it helped us.

There is no reason to look at this in any way adversarially. I guess it touches a nerve, because of the similar silly comments elsewhere, worrying about how the USA is "losing" if, say, the Chinese economy's size exceeds the size of our own. Other folks doing well doesn't make us worse. When it happens we should be happy: other people are doing well, and we've got an opportunity to learn and improve too!

I talked to a (very) senior scientist at the World Meteorological Organization not too long ago. He was of the opinion that the U.S. weather models were about 5 to 10 years behind on the European models, and that this was caused by structural underfunding over longer periods of time (i.e. past decades). As I understood it these weather models are basically humongous software programs that are developed in house, but based on publicly available science and fed by data from weather stations.

Does your software become better if you run it on a faster computer? It will certainly help, but he instilled on me the impression that the US had a bit more catching up to do than just buying a new supercomputer.

He is correct. The European model still outperforms the US (GFS) model: https://twitter.com/RyanMaue/status/700216724067586049

I believe this reflects the community consensus. Perhaps partly money, partly communication/coordination, partly technique. It's embarrassing, and comments nearby saying "we have something to learn" are too complacent. US predictions over CONUS should be at least on par with ECMWF predictions.

You are correct, and I don't think there is much more of a collaborative than a competitive spirit between the various organisations. However, it is not only the model that is important in weather forecasting, but also the data assimilation system. This is an active area of research with a lot of "art" involved, or, perhaps more realistically, very educated guesswork, about how best to set up and tune any given system. ECMWF have been developing their 4D-VAR system in the atmosphere for quite a long time now, and it is very successful. I think the Americans might be using an ensemble Kalman filter, which has a whole range of tuning issues, so that will probably take them a while to get right. There is an interesting paper that interprets the effects of these two systems that I still don't fully comprehend [0]. The Met Office also uses a 4D-VAR system, but I'm not really clear on the key differences between it an the ECMWF system.

Another important topic is coupling ocean and atmosphere models, and how you handle the data assimilation across your coupling - again, this is an active area of research, with lots of subtlety.

Of course, the model dynamics and physics are also very important, as is the resolution. This last issue is one of the places where bigger computers have really direct benefits, the other being increasing the size of an ensemble.

It's also important to realise that the various systems tend to be better suited to certain things, so whilst the ECMWF system is better in key global metrics, it doesn't (on average) provide better forecasts of all quantities in all situations.

[0] http://onlinelibrary.wiley.com/doi/10.1034/j.1600-0870.2001....

And even with the new computer the European model is still better at mid range forecasts.

Great that there is a more powerful computer helping predict the weather! One should know that weather is a chaotic system and after three days its very hard to predict the weather correctly. This is known by the lorenz equations.

http://www.uvm.edu/~cdanfort/research/danforth-bates-thesis....

Here is the UK met office. "Most of the time the atmosphere behaves rather like the lower-left picture where we can predict with confidence for a few days and have to use probabilities thereafter. " http://research.metoffice.gov.uk/research/nwp/ensemble/conce...

Indeed! though the constraints that chaos imposes on predictability is a subtle question. See, e.g., ftp://mana.soest.hawaii.edu/pub/rlukas/LSASI/ENSO/Prediction/Predicability_a_Problem_2006.pdf

Maybe it's me, but having the backup in Orlando sounds like a bad idea. Anyone able to comment on the logic behind this?

Florida, for obvious reasons, cares a lot about weather forecasting and is likely more willing to pony up some of the costs, infrastructure, staff, etc. to support this.

Meanwhile, Orlando itself is relatively safe when it comes to hurricanes. Few make it far enough inland to cause significant damage there. (Though, obviously, some do. I was there during the 2004 season.)

I'd assume that one of Florida's senators, Bill Nelson or Marco Rubio, was responsible for ensuring that the backup system landed in their state. One person's pork is another's 'crucial jobs and infrastructure earmark.'

Could they have built it underground?

"Underground" isn't very feasible in Florida. Most of the state is effectively a sandbar with a very high water table. Underground construction is uncommon.

"Underground" isn't very feasible in Florida.

Here in central Texas, we have the same thing, but for the opposite reasons. Our water table is pretty deep - typical wells in my town are 900ft deep. But rather than sand, we've got limestone. Typical land has inches of topsoil sitting on top of solid limestone, so excavating a basement is prohibitively expensive.

>Typical land has inches of topsoil sitting on top of solid limestone, so excavating a basement is prohibitively expensive.

So you're telling me one could blast out a cave in the backyard? Spare no expense!

And even a slight sea level rise may make large parts of Florida effectively uninhabitable due to repeated flooding. If you want to build a reliable computing facility there then put it on stilts, not underground.

http://www.newyorker.com/magazine/2015/12/21/the-siege-of-mi...

Orlando is 82' above sea level.

If you're worried about it flooding, you should probably spend less time building stilts and more time dealing with the fact that Delaware no longer exists, nor does most of New Jersey, New York City, large swathes of the Eastern seaboard, 75% of Louisiana, the Texas coast all the way in to Houston, much of Los Angeles, and all of Sacramento.

On the other hand, it will be nice to splash in the ocean at Joshua Tree National Park, and I'm sure Arkansas residents will be happy to finally be able to own beachfront property.

I was in the south Everglades last week. We went over a pass that was _1_ ft tall. Coming from Colorado, the sign advertising the "pass" was just bizarre.

For the similarly curious -- I assume this is referring to Rock Reef Pass:

http://www.jessstryker.com/national-parks/everglades/rock-re...

That's it! Thanks, the picture was on my gf's phone and I couldn't reference it.

Is there anyone here who works on weather models who would be able to explain why custom supercomputers are still needed vs. large commodity clusters?

Thanks!

Modern supercomputers essentially are large commodity clusters. Nobody builds a single fast machine ala Cray-1 anymore. It's just not scaleable past a certain point and we're long since past that. However, they still like to give them fancy names as marketing, because they cost tens of millions of dollars.

The main differentiation vs a standard datacenter is low-latency high-throughput interconnects, often with a specific or reconfigurable topology. Something like InfiniBand, where you can ensure very consistent, very favorable bounds on your latency and throughput. So think of it as being a "custom installation" or "custom cluster" instead.

Nowadays there really are only a few real choices left in high-performance computing hardware: CPU vs GPU, processor architecture (eg POWER vs x86), and interconnection. Everything else is more or less irrelevant.

I have to admit though, I've always wanted to see what you'd get if you built a Cray-2 with GPU chips (eg Pascal) or Xeon Phi as processing modules. The sheer density of chips Cray managed to cram in that case is impressive.

Does this count, just curious. It is a large single machine not cluster. https://www.sgi.com/pdfs/4555.pdf

A cursory glance at that PDF suggests the exact opposite of your claim.

I was asking a question, not claiming anything. Thanks for your answer.

What kind of latency do they expect from these things? Less than 50 microseconds for a small RPC or small RDMA access? I'm wondering how far these things really are from commercial cloud clusters.

For a relatively modern cluster the standard is FDR Infiniband, which gives you 56 Gbit/s throughput. The latency for a small (< 32 kbytes) packet is less than 10 microseconds, down towards 1 microsecond for a 8 byte packet.

Glenn Lockwood has a nice blogpost with much more detail: http://glennklockwood.blogspot.com/2013/05/fdr-infiniband-vs...

Less than one microsecond.

https://en.wikipedia.org/wiki/InfiniBand#Performance

For more on Cray's Aries interconnect used on the Cray XC-30 referred to in the OP, see this paper from NERSC, which has one: https://www.nersc.gov/assets/pubs_presos/NUG2014Aries.pdf

Depending on topology, you can get ~1 µs latency with Aries. It's based on PCIe-3, so it's different from InfiniBand.

There is no such thing as a 'large commodity cluster.' You can order a pile of commodity parts and spend the next five years fiddling with it as you watch your metrics fall into the toilet, or you can contract with a systems integrator with domain experience to make sure that you can actually accomplish the work you set out to do. When you're moving an established set of software from a known environment into a new cluster, it helps to have a wide array of technical experts and SLAs in place to ensure that the environment -- here meaning the physical compute infrastructure, storage service, and software installation -- is reliable and maintainable, so that your team can focus on their core mission -- in this case, the modeling software.

"Chaos monkey" works fine when you're dealing with random cloudstuff and you are in a position to duck-punch your production code. When you have a large dataset that you need to process reliably and predictably, the mantra shifts to "fail never", and that requires engineering up front in combination with carefully-planned maintenance.

Full disclosure: I've worked on this specific supercomputer. While I'm not on the admin team in question, I do work with them occasionally and I have a lot of respect for them -- their job is not an easy one.

I work for the company that provided the storage for this. Our product is not a collection of commodity components but instead something very much more like an "appliance" that provides the performance at scale that NOAA needs. In general, one of the trends we see/fight is the "Hey, we can get components ourselves and use Lustre to build what we need" And.... they are spot on right. What you need to consider doing this is the amount of maintaining and administering that comes with this solution. The trade-off we see is that some groups, "Academia" for example, will prefer the cheaper solution because they have Grad Students, Post Docs and Undergrad research folks to help make the people cost free. Others, who just want to buy something that works in their infrastructure without adding the people costs would prefer to buy something ready made that "just works"

Bottom line: If you are using your system to get work done, the trend is to spend the money for reliability and availability. If you want to save money, it comes at a price (but is certainly do-able)

Related: Look at Apple, Google, Facebook, Amazon, Microsoft... they are large enough to build their custom systems themselves. Not everyone operates at this scale. Individuals can buy a cheap PC and build a NAS... or they can buy one from Netgear, Buffalo, QNAP that "jsut works" at a price premium.

I think the comments nearby about high communications bandwidth required by dense systems of equations are true -- basically they distinguish a tuned, integrated system sold by a vendor like Cray from a row of racks with commodity systems.

The point I wanted to add is that ECMWF, generally acknowledged to be the leader in NWP, also uses a pair of large Cray supercomputers (http://www.ecmwf.int/en/computing/our-facilities/supercomput...).

Note that the Cray XC30 uses Xeon processors and a custom interconnect. The old ECMWF system used IBM Power chips.

Yes, it is the interconnects that are important. The UK Met Office has also recently upgraded to a Cray XC40 [0], and there are other XC30 systems in the UK NSS [1] and the US NERSC [2], both of which are used for weather/climate models, amongst other things.

[0] http://www.metoffice.gov.uk/news/releases/archive/2014/new-h...

[1] http://www.archer.ac.uk/about-archer/

[2] http://www.nersc.gov/systems/edison-cray-xc30/

Just a guess...

high fidelity weather modelling algorithms are highly constrained by node-to-node communication latency. So their supercomputer probably has some networked-direct-memory-access or another non-standard setup tuned toward passing data between adjacent nodes quickly.

The branding is still very valuable. This article makes them sound like common blades, ready for virtualization or containerization: http://m.theregister.co.uk/2014/09/30/cray_adds_haswell_and_...

I don't do weather, but probably it has a lot to do with inter-node bandwidth.

What's the biggest bottleneck for better forecasting: better observations, better models, or better supercomputers?

Cliff Mass also blogged [1] about the NOAA supercomputer. While a day after the USA Today's article, many folks appreciate his perspective. He's posted on this several times in the past year [2].

[1] http://cliffmass.blogspot.com/2016/02/the-national-weather-s...

[2] https://www.google.com/webhp?#q=supercomputer+site:+cliffmas...

Even though the US now has more computing power, the Euro model is still doing a better job at forecasting: https://twitter.com/RyanMaue/status/700216724067586049

For a newbie in this domain, how do we start doing our own toy forecasts (lets say for Bay area)?

A really nice intro is this classic paper on numerical integration of the equation for barotropic vorticity:

http://mathsci.ucd.ie/~plynch/eniac/CFvN-1950.pdf

It's something you can solve in a few lines of python and run with minimal compute resources. A nice thing to try is to feed in some data from somewhere like below and watch what happens as you integrate it forwards in time.

http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalys...

> For a newbie in this domain, how do we start doing our own toy forecasts (lets say for Bay area)?

There are some available weather modeling packages, such as WRF[0], as a mesoscale (few km to 1000 kms) model, that one can download and run. You'd obviously need to find a resource for the appropriate input data. I don't know how easy it is to get up and running, though.

[0] http://www.wrf-model.org/index.php

Well, this is happening just in time for all the weather patterns -- as we know them -- to change due to global warming. I guess it'll be not unlike understanding the universe according to Douglas Adams...

for anyone interested, this paper has a nice background on numerical weather prediction

http://www.elsevierscitech.com/emails/physics/climate/the_or...

I got about 3 pages into this and felt like I was following the discussion until the discussion of an incorrect prediction result that:

> In fact, the spurious tendencies are due to an imbalance between the pressure and wind fields resulting in large amplitude high frequency gravity wave oscillations.

Since this paper was published within the last 20 years, I can't imagine what they were referring to by 'gravity waves'

Do you know what is meant by this statement?

Gravity waves are waves in a fluid that obtain their restoring force from their buoyancy relative to the surrounding fluid. See [0] for the gory details.

Some examples of gravity waves in the atmosphere: [1] [2]

[0] http://glossary.ametsoc.org/wiki/Gravity_wave [1] http://cimss.ssec.wisc.edu/goes/blog/archives/2051 [2] https://www.youtube.com/watch?v=yXnkzeCU3bE

It's funny that you point this out, since some people in the field of astrophysics are quick to correct others when "gravity waves" are used when they really mean "gravitational waves", and sometimes vice versa.

Basically, gravity waves (or g-waves) are a type of perturbation in stratified media where the restoring force on sound waves is buoyancy. The other scenario is where the restoring force is pressure, which are pressure waves (or p-waves).

Gravity waves are important in not just planetary atmospheres but stellar media as well, particularly in the outer region of stars. A popular candidate for /gravitational/ wave sources is binary neutron star systems, so the two words aren't interchangeable there since they refer to two very different phenomena!

Are you mixing gravity waves and gravitational waves? Gravity waves are any waves in media in which gravity acts as restoring force, like waves in ocean, or waves between layers with different densities in atmosphere, or, AFAIU, whole atmosphere (think of atmosphere as a big shallow pool, with air sloshing around in the pool instead of some liquid).

https://en.wikipedia.org/wiki/Gravity_wave

> "Mass was highly critical of the federal government's lagging computing capacity in recent years.."

Perhaps he should have been having a chat with the NSA boys...

Wish they would stop using the library of congress' collection as a reference...

I know it's big.. no idea how big.. it's not tangible to me.. never visited.. I'm sure most americans have not visited.

You do not get an idea of how big it is by visiting; the stacks are off limits and much of the material must be ordered in from suburban warehouses.

They probably also have most of the bookshelves on rails? The local uni library keeps their old books on a system like this, it's really quite cool. The room can fit, say, 54 bookshelves tight-packed with no room to enter anywhere. So they put 52 shelves in on rails, and between each pair there is a wire going across. Disconnect a wire between two shelves, and all the shelves slide apart so you can enter where you opened the wire.

I'd estimate most university libraries have a system like this, though the "disconnect a wire" part is strange. Usually they're just a button on the shelf.

I think the wire is a safety feature to avoid accidentally crushing people. If you try to disconnect a second wire, nothing happens, and you have to go find the first disconnected wire and reconnect it. Then you will see if there is still someone in there. (When you're finished in a row, you're supposed to reconnect behind you when you leave.)

This helped a little:

"The Library of Congress is the largest library in the world, with more than 162 million items on approximately 838 miles of bookshelves. The collections include more than 38 million books and other print materials, 3.6 million recordings, 14 million photographs, 5.5 million maps, 7.1 million pieces of sheet music and 70 million manuscripts."[1]

[1] https://www.loc.gov/about/fascinating-facts/

Reply on news.ycombinator.com