A Simple Pattern for Jobs and Crons on AWS

An alternative would be to just use AWS lambda:

> You can create a Lambda function and direct AWS Lambda to execute it on a regular schedule. You can specify a fixed rate (for example, execute a Lambda function every hour or 15 minutes), or you can specify a Cron expression.

http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-e...

It would remove the need of infrastructure for "task workers". Although, depending on your stack / setup, it of course may be easier to have your application tier double as a task worker due to complicated dependencies or startup times.

Additionally, I am not sure if everyone is aware but SQS does offer FIFO delivery with guaranteed once-only "processing". There are some interesting caveats to this, definitely worth looking into though :)

I've used SQS for a simple job queue before. It works well and it's nice not to have to worry about more infrastructure. To make things even better, Celery now has an SQS broker, which I'd like to try out.

I've also had success with an HTTP subscription in SNS for simple async jobs. Having your tasks be just another HTTP endpoint is really convenient. It's not actually a task queue, but it's often a good-enough substitute.

One frustration I've had with it is that Amazon signs SNS messages, but I can't find a Python library for validating the signature. Not even AWS's flagship API client (boto3) has the functionality.

As simple as this article is trying to make things, it still feels like it's trying to re-create what elastic beanstalk workers already do out of the box... http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-...

You could also use `sqsd` (an Elastic Beanstalk worker environment) and avoid having to write any queue reading code.

Can't wait until Google Functions[1] adds the ability to schedule and define recurring function calls.

[1] - https://cloud.google.com/functions/

Something similar, and perhaps even simpler: Make a few SNS topics that lambdas can subscribe to - either one per method or one per method type, and then filter more in code - and push notifications to those from your service layer as needed to invoke async workflows. I've been doing this recently and it's pretty pain-free.

Been meaning to look at Step Functions for a lot of this async work stuff, but haven't had time yet (plus it's nice to let these things mature a bit before diving in).

For jobs that take longer than Lambda will allow, I use Datapipeline to run a ShellCommandActivity, which can run a bash script on an EC2 instance.