I believe this is the problem that Storm was supposed to solve ( https://github.com/nathanmarz/storm ). Or Amazon's SWF ( http://aws.amazon.com/swf/ ). They work in the case where you can define the flow between all of the tasks.