Async Task Queues

I think a task queue should be a default part of the web stack in exactly the same way that memcached is - once you have one set up a huge range of problems become things that you can quickly solve using a queue, which is good news for performance, scalability and maintaining a clean architecture. The initial setup costs pays off very quickly.

Part of me sees things like this and my inner curmudgeon wakes up and yells "This is what happens when you use things that are new but not better! This is standard knowledge that has been around for 20 years! If you only used a _real_ web language/stack you wouldn't be reinventing everything! JMS anyone?!"

However, my better angels re-read it and say "Well, at some point you didn't know all that, and this is actually a well-written intro to the topic".

So ... good article!

Task queues also allow for interesting web application architecture when combined with technologies that enable the server to push events to the client (think WebSockets, Socket.IO, etc).

The potential is far greater than simple tasks like sending an email, fetching external data, etc. If the server has the ability to push updates, you can start to use task queues in core user interactions. This is possible when workers can notify clients when their task is complete.

For example, you can have clients submit HTTP requests to the web server, the web server dispatch jobs to the task queue, worker processes (potentially in a different language than your web server!) retrieve those jobs, and then the workers directly notify clients when the job is complete. Combined with client-side logic, the result can be a good user experience, even when the server is performing complex tasks.

For example, I'm using this strategy to write web-based statistics software (think web-based Stata or SPSS). Without this type of asynchronous architecture, it simply wouldn't be possible to provide a good user experience because the core things the user wants to do are computationally intensive.

I'm by no means versed in async task queues, but Celery (https://github.com/ask/celery), a python task queue, has worked quite well for me. Interesting article.

Amazon's SQS (http://aws.amazon.com/sqs/) is a nice queue implementation with a very simple API for widely distributed systems. See the boto Python AWS library documentation for example usage: http://readthedocs.org/docs/boto/en/latest/sqs_tut.html.

Google's App Engine task queues are also very easy to use as the workers are just normal URL handlers in your app and the payload is standard HTTP data. See Python documentation at http://code.google.com/appengine/docs/python/taskqueue/ and Go documentation at http://code.google.com/appengine/docs/go/taskqueue/.

It would be really cool if we could compile a list of tasks on a webserver that these task queues could carry out.

The article lists: Image resizing Sending emails Third Party API calls

What about? Database optimisation Static file compression

At LayerVault, we're using delayed_job (https://github.com/collectiveidea/delayed_job) for our queuing system. Everything goes through a database, so the performance won't be as good as something in memory. Also, I feel safer with the workers only communicating with queue servers and not with application servers. Just a litte more abstraction :)

Storing the queue locally is great for speed, but it sacrifices durability. When an app server crashes, all those queued tasks are lost in the ether.

I once built a task queue using redis, start-stop-daemon and php. start-stop-daemon can turn vanilla php scripts(or any kind of script for that matter) to a worker.