Process Partitioning in Distributed Systems

miyagijournal.com

"Using queues also exposes you to the noisy-neighbor syndrome. If you have 25 jobs on the queue, all consisting of the same resource-intensive work, your queue could get bogged down while less intensive processing is blocked. Queues represent a false form of parallelism and are subject to such congestion."

This isn't a failure of queueing. This is a failure to use queueing correctly. You either haven't defined the proper queues, or you are failing to use priority.

Overall, I'd say the process partition model wins best when you can take advantage of sequential access paths to your dataset. In that case, don't enqueue work items for each individual data key - enqueue a work item for a range. Queues are fundamentally an excellent mechanism for managing work items.

Also known as: embarrassingly parallel problems.

Often true, but not exclusively so. One can sometimes create a process partitioning solution to a problem by exploiting speculative execution. You start workers which speculate the previous execution will result in their being run. If that speculation turns out to be false you discard their results, it if turned out to be true you continue with them. There was a talk at ISSCC about doing speculative branch prediction this way, two compute units where one proceeeds as if the branch isn't taken while the other proceeds as if it is, and when you finally get the result back that says which would have happened you retire the other thread, making it available for the next branch.

Reply on news.ycombinator.com