Hacker News Reader

Before GNU Parallel I used to use Ruby's workers and job queue to keep ${N} cores busy with work. It sorta worked like GNU parallel but was quite basic. I've since switched to using GNU Parallel. Stable code I don't have to write doesn't have to be maintained... not to mention it has more features than I normally supported.

View on HN

What did you use exactly? I am curious, Resque? Sidekick?

Ruby's Queue structure to push work, and Thread for spinning up workers based on the number of cores on the machine. Main thread would push all commands to run to the Queue, followed by ${N} shutdown hints, and ${N} Threads would pick them off in a while loop that would only stop when it saw a shutdown command. Once the last thread consumed the last shutdown hint, all threads were done and the script would exit. This was barely one step beyond a bash script that backgrounded all tasks at once and swamped a host until it slowly finished up.

Reply on news.ycombinator.com