Hi! I'm the primary author and maintainer of the library. Thanks for the interjection. The intended application for RaftLib is to make something that will scale, just as you mention. I wrote this post a long time ago when I was just trying to get people interested in using it. It's a simple example that shows you can take many lines of standard parallel code, and write a much easier to read (smaller) version in a very short time that performs just as well or better than the manually managed parallel code.

A long time ago I was a biologist, then bioinformaticist. I wrote some code that would scale to a single node, and to a few dozen cores. In doing so, I realized how much I hated writing the same boilerplate code over and over again. TBB, c++11 threads, OpenMP, and MPI all basically have the same level of boilerplate and gotchas. I wanted to make something that was relatively easy to use and easily integrable with C/C++ code. Go was the only thing that came close, but it was brand new at the time.

It occurred to me while working on the AutoPipe system as a grad student that I could do something even better than a simple coordination language and at the same time subsume the functionality of a lot of parallel libraries. With stream/data-flow processing, I can do the exact same things I can do with OpenMP and MPI, but I can do more. The state encapsulation allows a whole host of cool optimizations, like identifying bottlenecks and duplicating actors dynamically (there's a whole host of reasons we'd be limited in OpenMP, c++11 threads). You can also compile an encapsulated function to another hardware platform entirely, or use high level synthesis tools to go to an FPGA (I'll be going there again soon too with RaftLib). The only thing that has to be constant across optimizations, is the connectivity of the DAG. By maintaining a port interface, just like you would hardware components (see Arvind's work from MIT...he's famous enough I just have to say Arvind :), we can compose really complicated applications. The port interface, it turns out, is also perfect for distributed compute.

Awhile back, I also had the realization that iostreams were perfect for this paradigm. Once you get your head around the concept, it seems quite natural. If it doesn't take off as a library, oh well. I enjoy working on it, and using it so I'll likely keep developing it in my spare time.

In the interim, I'll get back to exascale hardware stuffs :).