Gearman, an open source framework for distributed processing

What was your problem with it? I have tried to use Condor for a couple things recently and was annoyed enough by it I concluded Gearman was probably the better approach, though I haven't used Gearman.

For the record, Condor probably works fine for its primary use case, which is long running single threaded processes on Unix that eat an input file and spit out an output file before terminating. I was trying to use it to have a bunch of processes on Windows that also occasionally write to a database. Was also trying to tag "slots" on specific machines to use certain installed dependencies. I advise against trying to use Condor and Windows, or on trying to use any of its advertised advanced features beyond the basic "standard" or "universe" model. Have had no end of trouble with the "stored credentials" feature that lets you run a process under a specific account.

I think a part of the problem there may be that a substantial percentage of the original user community are perl hackers, and tend to discuss their Gearman usage in the context of whatever else they're using - I see Gearman discussions reasonably often on the Catalyst IRC channel, for example.

Another reason is quite possibly that for the vast majority of tasks, Gearman "just works" - if the obvious way to use a piece of software works first time, talking to other users isn't particularly necessary (there's not really a GNU cat community either, for example :)

I'm not entirely sure what you'd do to fix that though (or even if there's anything except perception to be fixed)

I don't think you can see individual job data, but you (and/or your monitoring daemon) can telnet to the port and issue admin commands. e.g. 'status' will list all job types, with #queued, #running and #workers.

http://gearman.org/?id=protocol (search for 'Administrative Protocol' - 2nd hit)

Can never count Gearman out, and it is winning new user such as Instagram. Great for simple work queue. I recently switch from Celery to Gearman to reduce headache.

I was just using Gearman::Client and Gearman::Worker today... it's still used by at least me.

In short, Gearman is multi-language, multiple different backends, possible to do RPC-style synchronous processing. Resque and friends[1] are single-language, exclusively async, but also extremely transparent because they're just JSON blobs in redis. Resque has a huge following as well.

[1]: https://github.com/defunkt/resque/wiki/alternate-implementat...