Parallel Programming for C and C++ Done Right
speakerdeck.com
speakerdeck.com
The conclusions offered by this deck are mostly FUD.
First of all, Haswell, the architecture where those transactional memory primitives are available, isn't out for another year. Saying Knights Corner was available on November 2011 is also deceptive; Intel demo'd it at SC11, but you can't buy one yet.
Second, he helpfully glosses over that Cilk++'s elemental functions are identical to how CUDA and ISPC work; write a specially decorated single-threaded function, use a specially decorated function call, and end up with parallel work. I think it's exceedingly likely that the industry will standardize on this as the data-parallel methodology of choice within the next ten years. That timeframe will depend on how quickly GPUs and CPUs converge in terms of functionality (with vastly different performance characteristics). Task-parallel stuff will be done with something else.
The really difficult question will be how to get performance portability. C++ (or Fortran) code that runs well on Haswell will probably run like crap on KNC and vice-versa due to differences in the number of threads you need in flight, cache sizes, vast latency differences, etc. (Look at OpenCL running on two GPUs or especially CPU vs GPU as an example today.) Solving that is going to be the real challenge.
Can anyone comment on the number and quality of "hard core" C++ jobs and what you think the trajectory will be like? Right now I use C++ almost exclusively at work but the code/concepts involved aren't too difficult.
C++ has two big advantages over alternatives like Java: very low and deterministic processing latency (important for real-time) and very efficient memory model (important for throughput). These are really both about memory, and C++ gives you detailed control in a way few other languages do. As to why this is important, memory performance has been scaling more slowly than many other aspects of computing architecture such that it is the bottleneck for a growing number of applications. C++ allows you to be very efficient with the memory architecture without much effort. If your application is fundamentally bound by memory performance, competent C++ can get you 2-10x returns on performance in real systems relative to languages like Java. (For some other tight-loop, CPU-bound codes, not so much.)
I'm pretty bull-ish on C++11. It is not so much that I am a fan of the language but that I am a fan of what it can do in terms of performance for databases and large-scale systems. Most of the high-scale and high-performance development going forward seems to be targeted at C++ these days. That was not always the case but the requirements of modern applications are somewhat forcing that choice.
The practice of a single process locked to each core communicating over a messaging interface has trickled down to more generic massively distributed systems work because it has very good properties on modern hardware. You end up doing a fair amount of functional programming in this model because multiple tasks are managed via coroutines and other lightweight event models. This architecture is very easy to scale out because it treats every core -- on the same chip, same motherboard, or same network -- as a remote resource that has to be messaged.
MPI has one significant problem for massively parallel systems in that it has tended to be brittle when failures occur, and on sufficiently large systems failures are a routine problem. There are ways to work around it but it is not the most resilient basis for communication in extremely large systems. At the high-end of HPC MPI and similar interfaces are commonly used but for many of the next generation non-HPC systems operating on a similar scale they are using custom network processing engines built on top of IP that give more fine-grained control over network behaviors and semantics. This is not faster than MPI and often slower, and tends to be a bit more complex but it allows more robustness and resilience to be built in at a lower level. MPI was designed for a set of assumptions that work for many classic supercomputing applications but which don't match many current use cases.
also, do you have any examples of custom network engines on top of IP?
The other thing that is increasingly important in the multicore CPU space is memory locality. It's vastly more common to be limited by memory bandwidth and latency than by the execution unit. When we start analyzing approaches with a parallel complexity model based on memory movement instead of flops, the separate address space in the MPI model doesn't look so bad. The main thing that it doesn't support is cooperative cache sharing (e.g. weakly synchronized using buddy prefetch), which is becoming especially important as we get multiple threads per core.
As for fault tolerance, the MPI forum was not happy with any of the deeper proposals for MPI-3. They recognize that it's an important issue and many people think it will be a large enough change that the next standard will be MPI-4. From my perspective, the main thing I want is a partial checkpointing system by which I can perform partial restart and reattach communicators. Everything else can be handled by other libraries. My colleagues in the MPI-FT working group expect something like this to be supported in the next round, likely with preliminary implementations in the next couple years. For now, there is MPIX_Comm_group_failed(), MPIX_Comm_remote_group_failed(), and MPIX_Comm_reenable_anysource().
At least within my company we haven't used much in the way of parallel programming languages like OpenCL/CUDA/Cilk, preferring instead to use a task based concurrency library not entirely unlike Intel's TBB, but more cross platform. There's more investigation into GPGPU happening these days -- I'm curious to see if it'll pan out for more than just rendering applications. I'm also curious to see if the other (physics/audio/animation) areas can pull GPU time away from the rendering guys :)
The trajectory in this field is predictable. You get a hardware platform, have 5-7 years to figure out how to use it, and once you finally have it figured out something entirely different comes along.
Admittedly, a lot of these jobs are oversees. However, I believe the amazon and MS consumer electronics groups are still in the US and I know for a fact that until this year all of the Toshiba TV firmwares were being done in the US.
However, my experience in the CE industry has been that we aren't worried about massively performant parallelism, just "moderately performant" parallelism. I want to say that as of a few years ago Toshiba TVs had something like 10-30 threads running at all times in parallel.
Since the process was reading, process (data parallelization), writing - and reading was much bigger than writing (you read big model, animation, or texture files, and write ~8x smaller), it was "parallelized" by prefetching files in the file cache reading them in a second thread in advance.
Allowing only one writer to the HDD turned out to be beneficial (for cases like RAID SSD you can do multiple writers efficiently but this is rather pricey).
Edit: nope, you probably meant CSPs (http://en.wikipedia.org/wiki/Communicating_sequential_proces...), sorry.
Intel removed the features that gave Cilk a reason to exist: inlets and aborts. They're useful for (partially) parallelizing hard-to-parallelize problems that don't fit well into the other frameworks like OpenMP etc. Why did they remove them? I'm guessing because they were difficult to do well, and the algorithms that need them don't present such nice linear scaling graphs for marketing slides.
However, without those features, Cilk just doesn't distinguish itself enough from OpenCL, OpenMP etc. Parallelizing easy to parallelize problems isn't the problem, it's the others we need help in dealing with!
The poster childs/demos for the original Cilk were parallelized chessprograms, some of which did quite well in real tournaments. It's a very well studied area that exhibits a lot of parallelism, but not in a form that's easy to extract (hence there are no competitive ones using OpenMP, GPUs etc). Cilk was able to do it, a major achievement. But you can't even construct those in Intel's crippled Cilk version any more. Well, not any that would be competitive, anyway, which is the point to begin with.
If your solution only solves the easy problems others have already solved, what exactly is your right of existence?
It came out of IBM (Eclipse license) at about the same time as Watson, and I've gotten the impression that it (compiled down to C++) was the main language used for it.
One of the authors was a guest professor whenever I took my parallel programming course and wound up teaching about half the classes, so its abilities and use may have been exaggerated slightly, but it has a lot of constructs built in that I'd imagine to be terrible to implement otherwise - good globally synchronized clocks and memory management across everything on the current "place" (roughly one physical computer), although you could still had to manage memory you sent to different places manually.
But Wikipedia says Watson's built mostly on Hadoop, where the coolest features wouldn't really have much of an effect, so it may be just a crazy research language. I was just curious.
X10 is also dying as IBM is shutting down its research groups. Unless some other institution (some university chair?) will step up for maintenance, its development will halt soon.
That is not a true statement.
To preprocess the corpus and create fast run-time indices we used Hadoop. UIMA annotators were easily deployed as mappers in the Hadoop map-reduce framework. Hadoop distributes the content over the cluster to afford high CPU utilization and provides convenient tools for deploying, managing, and monitoring the corpus analysis process.
The online parts of Watson use UIMA: http://uima.apache.org/