Simplifying Parallel Applications for C++: An Example using RaftLib (2016)
medium.com
medium.com
Not you specifically because you probably don't run applications on super computers with millions or even just hundreds of processors, but for those interested in scaling above multi threaded programs with Intel Threading Building Blocks, it could be an interesting library.
The article even points out that raftlib must be statically linked which forces a few kb to be used when the binary is loaded. In a super computer you would have a cluster of many machines and would need to load even dynamic libraries at least once per machine.
Then looking at the raftlib site [1] and their wiki on github [2] it is clear that they have no way to scale to super computers, yet. They do use the terms, like "execution kernels" and talk about abstracting where the execution happens, but the focus seems to be on syntax (which is a thing I suppose).
Looking at the TBB developer guide[3] it seems clear that either would take some custom coding to get working across multiple machines.
It seems clear to me that Raftlib and TBB exist in the same space, targeting the typical app or system coder wanting threads. A much better point to raise if you want to support raftlib would be its focus on syntax, which is often ugly with many threading libraries. But the the raftlib guy thinks his looks good. I won't comment on something so subjective.
[1] http://www.raftlib.io/ [2] https://github.com/RaftLib/RaftLib/wiki/Execution-Model [3] https://software.intel.com/en-us/node/506045
> enables a programmer to assemble a massively parallel program (both local and distributed)
Mentioning distributed parallel program seems to indicate they do support clusters. The Wikipedia page could be wrong. If it supports multi processes on a single machine it should work on multiple nodes too.
Compared to TBB as you mention : [2] it's only focused on shared memory parallel programs.
[1] https://en.m.wikipedia.org/wiki/RaftLib
[2] https://software.intel.com/en-us/forums/intel-threading-buil...
EDIT: From the paper cited in Wikipedia :
> As to communication between nodes, RaftLib seamlessly integrates TCP/IP networks, and the parallelized execution on multiple distributed compute nodes is transparent to the programmer.
> The streaming compute paradigm generally, and Raftlib specifically, enables the programmer to compose sequential code and execute not only in parallel but distributed parallel (networked nodes) using the same code.
So it looks like it supports clusters at least over tcp/ip, but it's not hard to imagine another lower level communication protocol on a cluster.
That being said, one reason they only show threaded programs in the paper and in the example could be that the performance and scalability on many-cores is terrible. I wouldn't be surprised if it involves moving a lot of data across the network pipes.
On the multi-process..yes, that's easy. I've removed it for the current main-line branch given the lack of demand. IfDef'ing the code made it much easier to proceed with getting it ready for alpha. The FIFO mechanisms are well tested using SHM and the forking code will be added back in soon. Another thing I commented out to get it working on multiple platforms is the NUMA placement code, now that I think hwloc will work on all platforms I'll get it added back in. Helps out on cross-socket communication quite a bit, as well as placing buffers closest to PCIe root for data transfer to accelerators.
In reality the data movement is no worse than any OpenMP or other parallel program. In as many places as we can, the data is left in place vs. pushed. Between nodes, it gets more fun...however it's still a rather well understood problem. Thanks again for the interest! I'll see if I can do a ShowHN before CPPNow 2017 for the beta release.
> Between nodes, it gets more fun...however it's still a rather well understood problem.
Indeed it's understood, but the scalability problems switch somewhere else once your communication becomes the bottleneck. One advantage of dataflow-like paradigms like yours is that you can avoid major barriers like in Bulk-Syhnchronous-style programs. Those barriers start to become costly when you run with ten/hundreds of thousands processors. That'd be super interesting to see your library scale in those settings, similar to [1].
[1] https://www.researchgate.net/profile/Mani_Zandifar/publicati...
A long time ago I was a biologist, then bioinformaticist. I wrote some code that would scale to a single node, and to a few dozen cores. In doing so, I realized how much I hated writing the same boilerplate code over and over again. TBB, c++11 threads, OpenMP, and MPI all basically have the same level of boilerplate and gotchas. I wanted to make something that was relatively easy to use and easily integrable with C/C++ code. Go was the only thing that came close, but it was brand new at the time.
It occurred to me while working on the AutoPipe system as a grad student that I could do something even better than a simple coordination language and at the same time subsume the functionality of a lot of parallel libraries. With stream/data-flow processing, I can do the exact same things I can do with OpenMP and MPI, but I can do more. The state encapsulation allows a whole host of cool optimizations, like identifying bottlenecks and duplicating actors dynamically (there's a whole host of reasons we'd be limited in OpenMP, c++11 threads). You can also compile an encapsulated function to another hardware platform entirely, or use high level synthesis tools to go to an FPGA (I'll be going there again soon too with RaftLib). The only thing that has to be constant across optimizations, is the connectivity of the DAG. By maintaining a port interface, just like you would hardware components (see Arvind's work from MIT...he's famous enough I just have to say Arvind :), we can compose really complicated applications. The port interface, it turns out, is also perfect for distributed compute.
Awhile back, I also had the realization that iostreams were perfect for this paradigm. Once you get your head around the concept, it seems quite natural. If it doesn't take off as a library, oh well. I enjoy working on it, and using it so I'll likely keep developing it in my spare time.
In the interim, I'll get back to exascale hardware stuffs :).
'Storm, Samza, and Spark are open-source streaming platforms that are focused on message-based data processing. These systems differ from other classic "streaming" modalities in that they eschew point to point communication for centralized brokers to distribute data.'
That's maybe true for Spark, but it's not true for Storm and Samza (or of Streams). Yes, there is some centralization for application control, but there is no centralized data broker. That is, all messages are not routed through a centralized broker. I believe Storm does route all messages that come through the same JVM through the same logic and connection, but that's quite different from having a completely centralized broker. I can't speak for certain Samza, but I would be very surprised if it used a centralized broker.
In Streams, I can say for certain that there is no centralized broker; messages are definitely sent "point to point."
The benchmarked version also has a dynamically resizing FIFO which uses utilization of the queue itself to guide the sizing. This means that the FIFO can better adapt to dynamic behavior found in most applications run on top of an operating system (most all these days outside of HPC). Looking at load stalls, the RaftLib version has fewer, but not quite enough to account for the results.
If you look at the single worker thread case, then jump to two threads..you can see a fairly big jump. RaftLib by definition is a pipelined programming system. The read file and compress are done perfectly in parallel. The bzip2 code doesn't quite pull it off in a perfectly pipelined fashion. It's close, but not quite. This results in less overlap of execution and communication. If I'd run on Linux (thread affinity on OS X is well, fun last time I checked..if not impossible to do manually), I'd also add thread affinity to the list which most people don't bother to optimize. Hot caches and synergistic cache accesses are quite beneficial.
TBB was in alpha at one point. So was OpenMP, MPI, ...
Just because you have an established library doesn't mean something else can't come along. Said another way, just because new doesn't mean to dismiss it. <rant> Do you have a quill pen? Hmm, didn't think so. If somebody came along and said: "Hey, look they both write, why use that fountain pen." Then we'd likely be writing with feathers today. </rant>
That something else has to be unique in some way in order to justify its use over the established thing. The article didn't at all touch upon what RaftLib makes unique in comparison with TBB.
> RaftLib enables programmers to stitch together parallel code from sequential kernels (both pipeline and task parallel), with far fewer lines of code than most other parallelization frameworks (while being just as performant).
Never give up until you feel yourself that advancing is not going to work!
Looking forward to the beta and the full release of your lib! The C++ community needs this !