[1] http://arstechnica.com/information-technology/2013/11/18-hou...
* Using virtual machines isn't a bad idea for cases where you have a one off calculation for a paper that you need to publish. But for some groups they need to continually model and predict stuff. For example, weather forecasting needs to be done every day.
* Hadoop MapReduce and friends are extremely slow. Virtualization slows things down as well. The person in your article was using software called Shrodinger which isn't MR.
* Often, MapReduce performance is bottlenecked by the shuffle stage. This requires a lot of data to be passed around the network. Ethernet and http connections are not ideal for this if performance is your goal (though they are good for resilience). RDMA (remote direct memory access) with Infiniband is much better (offloading lots off the cpu, less chatter, etc). There exist RMDA plugins for Hadoop but they're of ymmv quality.
* MR isn't an ideal paradigm for every calculation. MR comes from search indexing (google) and is useful in areas where you need to look at all the data and moving the data around is more intense than the calculations. If you can filter, feature extract, or otherwise reduce the amount of data being moved around, the benefits of using HPC style clusters win out. Places where you must look at all the data: Indexing, Adversary detection (insurance fraud, intruder detection), format change (mp4->webm, gz, json->bson), ingestion, backups. If you're not doing these things you might want to consider whether MR is right for you.
* Even in search indexing, deep learning is becoming en vogue and this is extremely computationally expensive. This means the cost of moving the data around is again taking a back seat.
* A large part of the benefits of Hadoop style MR was having local data. But network speeds have outpaced drive speeds so data locality is less of an issue. Coupled with deep learning, there's less need for HDFS type systems. GPFS or Object storage is fine.
Also, the page says it's only a theoretical petaflop:
"While Stowe says the Amazon cluster hit 1.21 petaflops, that's the theoretical peak speed rather than the actual performance. In the Linpack benchmark used to test supercomputer speeds, the theoretical peak is always reported, but the real-world results are what count when ranking the world's fastest machines."
There've been several points where getting a detail right has seriously improved performance. These made me wonder how well the generic map reduce approach could really deliver on performance across use cases.
>There've been several points where getting a detail right has seriously improved performance. These made me wonder how well the generic map reduce approach could really deliver on performance across use cases.
It can't. A big selling point of Big Data systems is that you can analyse unstructured data that you haven't seen before and quickly dig in. But then if you try to reproduce the benchmarks you basically have to perform a parameter sweep to get the performance numbers that people are advertising. Maybe there are some super experts who can divine the parameters for their calculations ahead of time, but I've never been able to do it.
Then on the HPC side you have people who look at the Big Data benchmarks and then take is as a challenge by writing some specific code to tackle the problem which trivially smashes the Big Data solution. Usually using a clever encoding. But this misses the point -but doesn't really since I think everyone was cheating/not-cheating anyway.
[0] http://www.nrlmry.navy.mil/coamps-web/web/home [1] http://www.wrf-model.org/index.php
And fnord123, could you please email me?
While Mass is the most outspoken on the subject, many experts insist that if the Weather Service wants to meaningfully improve its predictions, it must employ a technique called ensemble forecasting. The basic premise is either to tweak the physics equations or to make repeated changes to a model’s variables: You might bump up the temperature slightly, for example, and then run the model again. After a half-dozen or so reruns, you get a set, or “ensemble,” of forecasts that can be compared with one another. When all the forecasts in an ensemble agree, it’s a reasonably sure bet that the predictions will pan out.
Nobody I’ve spoken to doubts the superiority of ensembles. Yet they haven’t been widely adopted in the United States at the resolution required to forecast localized, or “mesoscale,” events — specifically, thunderstorms, flash floods and tornadoes — because high-resolution ensembles require more computing power than the National Weather Service can currently provide. Higher-resolution ensembles translate to greater accuracy in the same way that HDTVs are clearer than analog sets. I met with a scientist at the National Center for Atmospheric Research in Boulder who showed me a prototype mesoscale ensemble for the United States. But at the moment, he can’t exploit its full potential because the supercomputing cluster at the Weather Service simply couldn’t handle the load.
HPC systems are essentially designed for this type of computational problem, and we still want more powerful versions of it. For this reason, I find it unlikely that commodity cloud infrastructure - which was not explicitly designed for this type of problem, and it almost assuredly going to perform worse - will have much success in this area. At least in the near future.
So the distinction of "commodity" that Google made in the original map reduce paper in 2004 is gone. And the benefits of data locality are all but gone. The paper was great when it came out but it's long in the tooth since the hardware of today is different.
To answer your actual question, I've never heard of anyone running WRF on cloud infrastructure. It doesn't have to be a terrible idea. If you're trying to come up with a competing model then the agility you get might be needed. i.e. being able to roll onto new hardware as it becomes available instead of having to work with budget cycles for a hardware refresh lets you move from tesla k80 to titans whenever AWS decided to make them available.
Searching around, I see Glenn Lockwood did some benchmarks in 2013 with AWS:
https://glennklockwood.blogspot.be/2013/12/high-performance-...
This lot seem to have done some more benchmarking in 2014 (tldr 44% overhead on VMs - but take the time to read it since glockwood is always a good read): https://www.researchgate.net/publication/301407792_Performan...
I think the prevailing issue is that latency on virtualized nodes is poor so you have to work around it. Maybe fat nodes with gpgpu can overcome the issues.
If you need to talk to me privately then you can message me on reddit.
Abstract is here: http://www.eresearch.org.nz/event/eresearch-nz-2013/do-hpcs-...
They ran it for 18 hours. So that's $44k/day for 1 petaflop. This supercomputer is 130 petaflops. So let's pretend it scales lineally.. that's $5.7m/day.
In just 30 days on AWS, you could buy the $173m supercomputer in the article.
This isn't a serious alternative.
For highly parallelizable and distributable applications (anything where mapreduce makes sense), commodity clouds are awesome. Just like relying on the subway in a city. You'll have a bit of a hassle when you need to go shopping, but your normal use case is such that it is worth the hassle.
Now let's say you have a code where your tasks require much tighter coupling due to communication (basically "Science codes"). In these cases, what you want is the interconnect. You want to be able to get a physical block of nodes in a very short amount of time for a somewhat long amount of time. In terms of cars, imagine that you live outside of a city or tend to do a lot of shopping or long distance driving.
And that is basically the logic here. It sounds like the Japanese rig is intended to be similar to the US's ORNL Titan in that it is primarily a capacity machine for science in general. So people with those kinds of jobs can buy time and run their large simulations with ease while still relying on stuff like AWS for postprocessing and the like. In that case, even though this is a 130 petaflop machine, it will likely only ever run one or two jobs at that scale and will instead operate as a "cloud" with a focus on large contiguous allocations.
Whereas the other form of SuperComputer are the ones meant for large jobs for a small subset of users. Those are the ones that tend to be owned directly by governments and involve a fair amount of effort to get access to. Those run a mixture of the above kinds of jobs (for the target community) as well as actual large scale simulations. As for what they are simulating: Think about what would need really detailed simulations using codes and data that probably should never be near Amazon's servers.
I've pointed this out to people in SC community, they don't really care. Either they have a problem that only runs on a SC (IE, a science code with strong scaling) or they're just not interested in trying other approaches.
Anyway, a supercomputer for deep learning makes sense. The communication patterns in deep learning can be very intense, with a lot of n2 calls, and so a good interconnect might speed up the code. However, tensorflow etc aren't designed for MPI SC architectures.
I think this will probably change: I think DNNs are going to run on supercomputer-like hardware, but not something that is a traditional supercomputer with MPI.
There is one paper showing a port of TensorFlow to MPI (WHYYY?) and the results aren't very good.
Newer Intel chips will be integrating low-latency interconnects so in the future, Amazon, Google might be adopting low-latency interconnect computing.
Anime rendering? pretty rad action scenes!