Using a Message Queue vs Background Processing. | Hacker News Reader

Using a Message Queue vs Background Processing.

mikeperham.com

14 pointsmnbbrown7 comments

I'm not sure that I fully understand why the author considers message queuing via something like ActiveMQ to be complicated or difficult.

1) Install ActiveMQ (or my preferred option, ActiveMQ Apollo http://activemq.apache.org/apollo/).

2) Write an email script that subscribes to the message queue, and sends an email for each message received.

3) In the web app, publish a "send email" message to the message queue.

That's it - nothing more complicated required. Look at the number of libraries that use the Stomp protocol, and you can be up and running inside of a few minutes.

Spinning off a separate thread to perform this work is entirely sufficient and much simpler

I strongly disagree with this advice, if only because it's so incredibly easy to screw up from a security perspective. If you want to handle errors or other unexpected issues, doing so via spinning up background threads is going to lead you to a world of hacky pain.

One of the most inspiring articles I read about using queues is 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you:

http://www.engineyard.com/blog/2011/5-subtle-ways-youre-usin...

The article you linked too is interesting because:

"Storing a list of messages to send: whether it’s emails, SMS messages"

seems to be something like reddit does [1] & [2]. Is this the case:

[1] https://github.com/reddit/reddit/blob/master/r2/r2/lib/email... [2] https://github.com/reddit/reddit/blob/master/r2/r2/models/ma...

Why? Why? Why?

Your local SMTP-based Mail Delivery Agent (MDA) can be configured to to deliver mail directly, or to forward to another host, a practice called Smarthosting, and let the smart host deliver all the mail.

So instead of installing a new queueing system, use the one that 's already there. Deliver your mail to the local MDA (e.g. open a pipe to sendmail), and let a time-tested mail queuing system do the job that it was designed to do.

tl;dr:

You are doing bg proc or MQ wrong. Use my threaded worker

https://github.com/mperham/girl_friday

Does anyone know a Django or WSGI equivalents of this? I am having the same sending mail problem, and I really do not want push serialized data to another process via a complex & inefficiant MQ.

Take a look at Bridge (https://www.getbridge.com). You could use that to create a worker responsible for sending mail. Use Bridge to asynchronously route emails to that worker. When you need to scale your send mail worker, just fire up more instances of that worker and Bridge load balances and scales your worker queue.

https://bitbucket.org/pmclanahan/django-celery-email

everyone is wrong, everyone is right. just use what tool you know best.

resque, or beanstalkd or zmq push/pull whatever

Reply on news.ycombinator.com