My RabbitMQ setup for notifications | Hacker News Reader

My RabbitMQ setup for notifications

andreim.net

105 pointsandreimarcu17 comments

There's a cool and important feature to this setup that makes RabbitMQ specifically a great choice: queuing one message sends the notification to all devices at once.

In RabbitMQ, messages are published to exchanges and based on the type of the exchange, messages will be routed to specific queues, which is where the messages are consumed.

There are three types of exchanges you can publish to: Direct, Fanout, and Topic. In a Direct exchange (which is the default), a queue subscribes to the exchange with a routing key and any messages delivered to the exchange with that routing key is delivered to that queue to be consumed. So in a sense it's 1-1: exchange -> key -> queue.

However, in a fanout exchange, every queue subscribed to an exchange gets a copy of the message. This means that when you deliver a message to a fanout exchange, that exchange will give a copy of that message to every queue that is listening.

So you can send the "New Item" message to the "notify" fanout exchange, and every queue bound to that exchange will get the "New Item" message; i.e. the phone notification service, the web client notification service, etc, will all get the "New Item" message and can react accordingly. What makes this even better, is that if you add a new service (e.g. a new iPhone app), all you have to do to tap into your notification system to receive new item alerts is to tell a queue to listen to the notify exchange! You hear me right: you don't have to change any existing code.

Now, Andrei Marcu's aamnotifs is written in Python and he uses pika as his Rabbitmq module. Pika is a little more relaxed about the exchange types: any queues subscribed to an exchange with the same routing key will be treated as if they are attached to a fanout exchange, and will all receive copies of each message sent to the exchange with that routing key.

I say all of this just to point out the important point that Andrei left out: to add a second, third, or nth listener to your notification queue which receives a copy of the updates, you just have to add another

    n.receive("routing_name", print_notification)

line and change print_notification to whatever method you want the message to go to. For example:

    import notifs

    def print_notification(title, message):
        print "Notification received: {0}: {1}".format(title, message)

    def web_app_notify(title, message):
        print "Webapp notification received: {0}: {1}".format(title, message)

    def iphone_app_notify(title, message):
        print "iPhone App notification received: {0}: {1}".format(title, message)

    def android_app_notify(title, message):
        print "Android App notification received: {0}: {1}".format(title, message)

    try:
        n = notifs.Notifs("amqps://user:password@domain.tld:5673/%2F")
        n.receive("routing_name", print_notification)
        n.receive("routing_name", web_app_notify)
        n.receive("routing_name", iphone_app_notify)
        n.receive("routing_name", android_app_notifify)

    except KeyboardInterrupt:
        break

gist here: https://gist.github.com/rickhanlonii/6091542

Is it possible to have it so that it's fanout but a "die on first receive"?

i.e. If I send out a notification I don't care if it's my phone, my desktop or my laptop that gets the notification, but if one of those devices gets the message (race conditions not withstanding) then I don't want the other devices to get the message.

Or is it the case that if they weren't online they'd not be currently subscribed so they wouldn't naturally get it? In which case what happens if there are no current subscribers?

I was wondering about this also. I'm not sure if there is an acknowledgement system built into RabbitMQ, but it sounds like that is what would be needed for this to happen.

By default any message picked up from a queue by a consumer is removed (acknowledged), but you can configure the exchange so that it waits for an ACK from the consumer before removing the message. This gives the consumer the ability to return the message to the queue, but is dangerous because Rabbit will never stop waiting on the consumer to respond.

You could share one queue amongst several devices, and Rabbit will ensure that only one device gets the message (with proper acking on the subscriber side).

If a message gets sent to an exchange that has no queues attached, typically it will get sent to a dead letter exchange.

Edit: Most Rabbit client libraries have two different modes, one where you automatically acknowledge any message that is received, and another mode where you must explicitly acknowledge that you've received the message.

You can use a direct queue for this. Just have the phone, desktop, laptop, et. al. consume from the same queue. Rabbit gives out messages in a round-robin to the listening consumers, and the first to pickup the message is the only queue to get the message.

Then, let's say there's an error in the phone notification and you want to pass it back and let the next service try. This is where Rabbit's ACK system can come into play: you can setup rabbit to wait for an ACK response from a consumer acknowledging that the message was received and processed. If the consumer errors, you can return the message back to Rabbit and it will be given to the next consumer in line (which is the next consumer not busy).

If a service is not online, then they won't be consuming from the queue, so they won't get the message. If there are no consumers, then the message will wait in the queue until a consumer binds to consume the message.

Hope this makes sense!

Indeed, I had the impression that this will not be as clear, I'm going to edit my post to mention it.

Thank you!

Routing with Rabbit is extremely flexible, in fact, you can hook up two exchanges together (with a Rabbit only extension). You can also give messages a TTL, so that they'll automatically delete if not fetched.

So you can have one exchange for each user, and any device the user owns attaches a queue to that. Then, any exchange that a user is interested in only needs to subscribe to that one exchange.

Rabbit really is awesome. Even on top of that you can setup reply-to queues so that a publisher can wait for a response from the consumer. This is nice if you have, say, an image processing server and want to give it an image to process and block until you get a response that it's done.

Pika is a little more relaxed about the exchange types: any queues subscribed to an exchange with the same routing key will be treated as if they are attached to a fanout exchange, and will all receive copies of each message sent to the exchange with that routing key.

Actually that's how RabbitMQ works. The direct exchange will send messages to every queue bound to it if the routing key matches. As in: "SELECT q FROM queues where q.binding_key = msg.routing_key"

The fanout exchange will do the same but without the WHERE clause.

The topic exchange will do as the direct exchange but let's say that instead of an equals comparison it will do a match to accommodate routing key patterns for topics.

Of course RabbitMQ doesn't use SQL, this is just for illustrative purposes.

The thing about RabbitMQ is that it's so powerful and configurable that it scares a lot of people, but for trivial use-cases it is incredibly easy to get running. If you're using it as a message queue within your LAN (i.e. no direct external interface) then even the HTTPS configuration can be skipped, and all you need to do is create a user and an exchange and have at it.

Agreed. I use RabbitMQ on my iOS app's server (https://www.letsgohomeapp.com) almost exactly for the OP's purpose; I drop outbound notifications to app users on a RabbitMQ queue and have a separate service which draws from that queue, processes them & sends them off to the correct destination, be it an email address or an iPhone via APNS.

RabbitMQ works great for this & is honestly quite easy to set up and integrate with (was trivial to get up and running in my app's Node.js backend).

As a non-developer, I'm curious what others are using RabbitMQ (and similar) for.

Examples that quickly come to mind are push notifications for apps, large e-mail blasts triggered when $event happens, etc. What are some things that you use this type of a setup for?

(As I said, I'm not a developer (network engineer and Linux guy) and I'm trying to think of use cases that would apply to my environment (ISP).)

Lets say you have metrics for certain aspects of your system that you're trying to monitor. When they cross a certain threshold you could have a notification sent out.

We use rabbitmq to send push notifications to clients, rabbit has a stomp plugin that can accept web socket connections see http://www.rabbitmq.com/blog/2012/05/14/introducing-rabbitmq...

A standard example uses a topic exchange for logging. A topic exchange will deliver messages to every queue with a pattern matched routing key.

For example, you can bind three queues with routing keys where:

queue A = citical.log

queue B = warn.log

queue C = * .log

(Note that there should be no space after the wildcard) Then, when you publish a message to that exchange with the key "warn.log" it be delivered to queue B and C but not queue A. This is nice because you could have queue A sending you texts, calls, emails, bike messengers, etc when it gets a message; queue B just an email; and queue C just logging. Now, you're easily getting logging messages where they should go, at the same time.

Even better: let's say you want to add in a metrics system based on your logs. It's dead simple to tap into that log feed and get all of your log messages to process. Just add a new queue:

queue D = * .log

Now your new queue is getting copies of all of those tasty log messages as well.

Another quick example: at Updox we have an API that partners can call to send faxes. Part of this process involves converting the faxes to images for the user to view, building thumbnails, or otherwise heavy processing tasks.

Now, for scalability we could load balance more and more full blown web servers, OR we could pass the heavy processing tasks off to a Rabbitmq exchange and have a processing engine pickup those messages and process them. If we do the latter, then to scale we can just add another processing engine and tell it to listen to the exchange! As we grow, we can add as many processing engines as we need!

I think the question really is, what can't you use Rabbit for?

I just wanted to say RabbitMQ is an impressive product. Just looking at the features it supports (clustering, message types, federation, shovel) and the plugin ecosystem (STOMP+WebSockets, MQTT).

Reply on news.ycombinator.com