Hacker News Reader

Comment by latch | Hacker News Reader

I'm curious as to why Gearman vs something else, say RabbitMQ?

I currently use Gearman in my app (and supervisord), but I also looked into RabbitMQ. It was this comment on SO that made me go with Gearman.

I would say that Gearman is better for queuing "jobs" and RabbitMQ is better for queuing "data". Of course, they are both really the same thing, but the way it works out for me is that if you are trying to "fan out" work to be done, and the workers can work independently, Gearman is the better way to do it. But if you are trying to feed data from a lot of sources down into fewer data consumers, RabbitMQ is the better solution.

Source http://stackoverflow.com/questions/2283955/rabbitmq-or-gearm...

I tried both technologies in my app, and Gearman was eventually easier to work with. YMMV.

Unfortunately, I don't remember why we originally went with Gearman. We did evaluate RabbitMQ (or was that ActiveMQ?) and at the time there was something that we wanted to do that Gearman was better suited towards. That was a long time ago, and my memory sucks!

Or zeromq?

I don't think 0MQ existed when we made the decision to go with Gearman, but a 30-second glance tells me it would probably be well suited to build a job server on top of, but isn't exactly an out-of-the-box job server like gearman is. For one thing it's not immediately clear to me how you would submit jobs via zeromq for which workers don't exist yet or aren't currently connected. That's just a first impression though since I haven't dived into it very deeply, obviously.

I didn't mean to knock the decision about choosing Gearman. I was just interested in your thoughts on comparing the two.

That's correct that ZeroMQ isn't a server. I tend to describe it as a sockets layer that has messaging semantics. If a host isn't up, you can still send messages to the socket. It queues them up, like you'd expect a messaging system to do.

So this also means you can open a socket to a host that doesn't exist. Imagine you turn a server on with no clients up yet. A PUSH socket will block. When you connect a client, by opening a PULL socket, the message exchange starts.

These socket types can be inproc, ipc, tcp or pgm. inproc is an in process queue. ipc is basically unix sockets. tcp is... and pgm is pragmatic general multicast.

To build a broker, you'd probably use something other than PUSH/PULL sockets. You would use REQ/REP (request/reply) sockets. These sockets give you synchronous communication. The request sends a request and is guaranteed a response.

I think their doc explains the rest better than I could, I recommend reading this: http://zguide.zeromq.org/page:all

I think this description offers the gist of how REQ/REP, and their nonblocking counterparts XREQ/XREP, can be used to basically script together your ideal broker for many conditions.

When you use REQ to talk to REP you get a strictly synchronous request-reply dialog. The client sends a request, the service reads the request and sends a reply. The client then reads the reply. If either the client or the service try to do anything else (e.g. sending two requests in a row without waiting for a response) they will get an error.

But our broker has to be non-blocking. Obviously we can use zmq_poll(3) to wait for activity on either socket, but we can't use REP and REQ.

Luckily there are non-blocking versions of these two sockets, called XREQ and XREP. These "extended request/reply" sockets let you extend request-reply across intermediate nodes, such as our message queuing broker.

Reply on news.ycombinator.com