Memcached based message queues | Hacker News Reader

Memcached based message queues

3.rdrail.net

24 pointstjake8 comments

This is an interesting idea. You'd have to watch out for usage spikes that would end up dropping a large number of messages (kinda like paging... I suspect this gets worse quickly as the same message is sent multiple times). If you put some guards on the sending side, you might be okay (but with multiple senders this could get messy quickly).

I suspect in the end you will grow this to the complexity of one of the already mentioned messaging solutions. If you don't like them perhaps something like spread (odd licensing), jabber, or irc (my personal favorite) would work for you.

A pretty similar design was my first try for a simple whiteboard system but switched to another misuse of memcahced with prepend and append. Timing out of old messages is a nice feature of this solution. These are dirty hacks comparing to a "proper" solution but sometimes they are OK enough.

Beanstalkd has a memcached like protocol but designed for simple queues. It's a fine system but I'm missing the timeout feature. http://xph.us/software/beanstalkd

MemcacheQ - http://memcachedb.org/memcacheq/

Is this going to be part of ThruQueue?

I was thinking of it. I recently rewrote thruqueue to support a api similar to SQS (where messages are locked on read and if not deleted put back into the queue). Not sure of a clean way todo this with memcached

I really cannot stand the idea of making message queues that speak the memcache protocol. That's like having a RDBMS that only speaks HTTP.

Memcache is an impoverished language for messaging. Which is ok, because it's designed primarily for cache fetch and retrieval. Just like you wouldn't really want to go fiddling around with database cursors through HTTP (looking at you, freebase) you don't want to describe queueing and delivery of your messages over memcache.

And it's not like there aren't alternatives. Stomp is a perfectly acceptable protocol for simple messaging. If you need to do something fancy and complicated, there is also AMQP. Both protocols are supported by mature servers that are proven to run in highly available production systems. So why, other than a case of NIH, would a company want to spend time writing and supporting one off infrastructure like that?

It is interesting that a relatively advanced piece of software, message queues, which most run of the mill projects most likely have no need for, has an interface that is compatible with a simple, straightforward protocol that was designed for everyone and their dog. I mean, if you need message queuing, you most likely have someone on staff who can design, write, and deploy a protocol language (especially if they can write the queuing software also) and who doesn't need to use compatibility libraries.

Great, your memcached-compatible queuing service can leverage all those memcached client libraries out there, but somehow I don't think that client libraries are the hard part of the queuing service.

But what if you don't need a full RDBMS but all the nice characteristics of a REST interface? Think about CouchDB and friends.

Reply on news.ycombinator.com