Using Gearman For Distributed Alerts | Hacker News Reader

Using Gearman For Distributed Alerts

tech.backtype.com

14 pointsomakase13 comments

I would be very careful with gearman.

We tested it at Justin.tv and managed to cause the main process (gearmand) to spin at 100%, stop accepting connections, and lots of other things. Not quite ready for production use, especially if you need persistence and use the memcached plugin.

Also, the python client library is in pretty rough form. Our developers submitted a patch or two, but it still doesn't reliably handle errors, network outages, failover, or reconnects. You probably want all of these things in your job server of choice.

What did you end up using instead?

We're in the process of migrating to RabbitMQ. It looks pretty good, but we haven't really put it through its paces yet.

To clarify, Gearman is not meant to be used as a message queue, but instead it's a job server. The example in the post is an interesting use case (and convenient for us because we were already using it as a job server), but not really what it's designed for. I'll follow up with a post on how we use it to do work.

What is the difference between a job server and a message queue that requires a return queue?

Not much, but the Gearman default is to require a response. You can almost treat gearman like blocking unix pipes over tcp, mixed with a FIFO queue. In fact, it comes with commandline tools to do just that. You get the option of submitting a job without waiting for a response back too, though.

I suppose Gearman is more of an RPC broker with sync and async modes than an MQ, but the differences are mostly semantic.

Perhaps I'm getting hung up on the semantics. We use it more to farm out function calls, rather than passing content or delivering large messages. The one time I tried to use it for large amounts of data I did have issues with memory and latency.

For simple RPC-style use (millions per day) though, our server has been stable (no change in memory or latency) for months without a restart.

Perl or C server?

Are you now using Beanstalkd instead?

What kinds of jobs were you submitting? I've also run into problems using the python client handling errors, failover, and reconnects. But no issues with the C API, processing about 5 million jobs a day.

We used the C gearman server from gearman.org, and it was highly unstable and very slow (added an extra 10-100ms to a job, which was noticeable in the UI), and did other crazy things like return data from other jobs when the frequency got too high.

We've since moved to the original Danga Perl daemon (rather than invest time switching message queues), and it's faster, uses less memory, and more stable.

Next step is to move to a different MQ. In our application we don't need persistence especially, so I'm thinking zeromq.

I'm sure I must be missing obvious here but why are they not just pushing those mails out directly?

All that gearman buys them here is... a longer delay until the mail is actually sent.

You're right, it does increase the delay. It's more of an example of how Gearman can be used.

Also, I _hate_ the idea of running an MTA on 30 machines that shouldn't be sending mail.

Yup and a good sized one at that, working snippets are always welcome. I was just a bit confused due to the contrived use-case.

I don't share your concerns about the MTA (they are lightweight and it's usually helpful to know that sending mails won't block) but to each their own.

Reply on news.ycombinator.com