Celery 4.0

Joking aside, we use Celery in production for asynchronous number crunching. We build a web UI in Django that raises Celery tasks that run long number crunching jobs. When the number crunching is done we look at the report through our Django web UI.

It works well enough, though we wish there were better built-in task management. E.g. A built-in API to reshuffle tasks (e.g. for I/O resource balancing) would be nice. celery purge also seems too drastic in purging every task in every queue. Or maybe we're just doing it wrong.

Yeah - everyone should ideally be comfortable with all this stuff but I try and keep anything that's more complex than a pip install in a virtualenv to an absolute minimum.

If you are on AWS, you could just use something like their email service to fire off the request, then you don't need a queue at all.

You could add your emails to a db table and have a cron job consume them.

But for sending an occasional email? I've never really had a problem with just connecting to the SMTP server in the request/response cycle. If it takes more than a second to send then something is seriously wrong. You could use Mailgun or similar services which have their own queue for handling bulk sends and further reduce the likelihood of a problematic blocking of the web-server.

You can do such things as passing a database unique key, GUID, or file-path to the raw data on disk. Obviously, you will also need to engineer around that if you've got a distributed system. The tangential benefit is that you're not using a "messaging" queue or system for persisting or semi-persisting your image data. That's a big no-no as such systems are transient in nature and that often doesn't align with binary or image processing.

Base64 is for when you want to pass around binary in a text-based format. E.g. XML or JSON. But do keep in mind that because of the encoding format, converting binary data to base64 does increase the payload size by about 15% or so.