Amazon Simple Workflow - Cloud-Based Workflow Management | Hacker News Reader

Amazon Simple Workflow - Cloud-Based Workflow Management

aws.typepad.com

104 pointsjeffbarr31 comments

First thought: D'oh, why didn't I think of this?

New resolution: consider everything I do from a cloud based perspective.

My Experience: A few years ago I considered windows workflow and built my own simplified version for a project. WF tooling at that time was still evolving. The requirements were all over the place and I thought it might be prudent in this case to build something to expect the worst.

As Murphy usually does, the worst never came almost 5 years later. The workflow engine works well, and simply enough for a simple event based system. It's more of a routing system and I'm fine with it.

Where ASW catches my eye is the admin tools. Like most apps the admin tools are among the most complex parts of the app. In ASW, so far I'm recognizing a certain simplicity.. in a generalized sense for a workflow system.

If I tried to do the same in another workflow technology, I would almost certainly be running into more configuration upfront, and the admin tooling to create and manage workflows from a UI perspective would likely be either too basic or to overkill for what I needed.

That's the crux of generalizing a solution to a problem, it risks being so much bigger than we need that it adds overhead to what we were liking to simplify.

ASW initially strikes me as being much simpler. It could be useful to managing state simply across distributed applications, specifically in the cloud. Not sure how the pricing would work though, that seems like a lot of money to pay for logic gates and a state machine, but I could be trivializing.

Now.. just to think up a simple fun utility or project to try this out with.. :)

Using (and even researching) new AWS services has always been surprisingly frustrating to me.

You have to sift through buzzword-filled ad copy and corporate-speak to find any explanation of what the product is or how to use it.

At the risk of sounding like a Microsoft shill (I'm not), you can already have your cake and eat it if you use Windows Workflow on top of AppFabric.

It has an arguably better feature set (including extensive transaction, correlation, failover, load balancing support), is extremely robust and can be deployed on EC2 or your own infrastructure (or both at the same time - you can scale your workers up on EC2 for example). The learning curve to do something useful is about 2 days.

It's pretty cool actually.

> At the risk of sounding like a Microsoft shill (I'm not), you can already have your cake and eat it if you use Windows Workflow on top of AppFabric.

I've been researching/demoing App Fabric at my job for a few weeks now to possibly use for our new SOA layer, and that was the first thing I thought when I saw the SWF email from Amazon this morning. Of course anyone not working in IIS or Azure (ie Linux or AWS) would get no benefit out of App Fabric, and this could be an excellent alternative regardless (judging by the high quality of Amazon's other services I mean).

I really love this new tool. I think Amazon has some amazing technologies and services.

Why don't they hire a designer and awesome UX guy and fix their terrible interfaces?

I know most people use the API's, and I realise it's not really stifling their business, but I still don't see why a little more effort can't go there. I am sure there are a lot of people around the world that would be very happy for the console to have some more features and be a bit easier to use.

This has puzzled me for quite a while as well. The AWS Console isn't just awkward, it's lacking some of the features the API exposes if I recall.

Yep. It seems to be impossible to delete an SSL certificate for an ELB via the console, for example. You can set one up, but not remove it - for that you need the CLI tools.

This really sounds too enterprisey. It's unclear to me what the benefit is of using SWF, and lost my interest when it started listing business definitions.

"too enterprisey" = a service that large businesses can integrate deeply into their existing applications, Amazon can bill them for, and yet another compelling reason for companies with obscene amounts of money to dip their toes into AWS.

Look at this feature next to the AWS Storage Gateway, recently announced: these are features to get businesses using portions of AWS so that they demonstrate their reliability and cost effectiveness at a small scale. Once a business is comfortable dealing with Amazon then they provide compelling reasons to expand into more of their services (EBS booting + attaching Storage Gateway volumes to EC2 instances for braindead simple cloud compute migrations away from expensive, inflexible on-premises datacenters, SES, SQS, SNS integration with SWF, etc)

Never forget that Amazon is playing the long game.

One usecase is analytics workflows. It can also be used to build more elaborate machine learning models (say over data that is too large to fit in one box, and requires distributed coordination)

Look past that. This is super useful - high scale services often have loosely coupled services which pass data from one stage to the other and act on it based on various decision factors. Almost everyone I know invents their own mechanism for gluing these things together - the SWF seems super useful there.

Windows Azure (my former team) actually tried to do this with Azure Workflow Services several years ago, but that got killed around a year into it's existence.

Depends on your volume, of course, but seems expensive. 1 million jobs scheduled, 4 steps per job? $200. And you're locked into their API.

You can have 8 workers each polling an Amazon SQS queue once a second for $19/month.

Having had huge problems with their queues (3 minute lags, spurious duplicates) I'm wary of Amazon's polling-based API. SWF's long-poll feature is smart versus SQS, however.

I think this is meant to be a parallel to things like celery and gearman, i.e. a task queue. A table mapping between AWS services and what components they replace in a typical DIY stack would be helpful for a lot of people (i.e. SQS = RabbitMQ, SWF = celery, ELB = nginx, DynamoDB = Cassandra(?)).

Cerery doesn't do workflows with dependencies, just tasks. This is more like Storm: https://github.com/nathanmarz/storm

I wonder if Celery could still use SWF as its backend in the future? Instead of having to manage your own EC2 instances and databases for the Celery daemons.

Well celery does have subtasks that it can wait and join on (see http://ask.github.com/celery/userguide/tasksets.html), but I suppose that the logic to do such things would have to occur within a running task as opposed to SWF where such things are handled by the platform itself.

I'm actually in the middle of designing a Workflow-as-a-service for my current employer (based on jBPM 5).

However, this isn't Workflow-with-a-capital-W. It's not even a state machine. SWF is actually a distributed-processing task queue with persistent state and execution history. Which is pretty cool.

This is a great idea! Actually this may have just saved me a bunch of coding.

Amazon you're starting to rule the world and its a bit scary...but for now i'm ok with it.

Have fun with it, and let me know what you come up with.

This is Amazon's attempt at next gen PaaS. You can find more details here http://www.cloudave.com/17315/remember-next-gen-paas-and-aws...

Yep, your analysis is spot on. Good article

This looks very applicable to what I'm working on right now. I've built my own naive implementation of a distributed worker queue, using redis as the basic centralized queue. There are many edge cases where redis lpush and rpop will let me down, so as I build my tool, I know that I'll eventually have to move to an actual message queue.

I've stayed away from RabbitMQ, or even something like ASW, while developing this project of mine because they're a lot of work to integrate, on their own. ASW looks awesome but it looks as though it'd take all of my focus to integrate it. It's something I wish I could use right now but don't think I'll be able to until I've finished most of my development.

I guess I'm just saying that ASW looks complex.

I don't get it, can someone explain what problem it is trying to solve? By means of an example?

This is very cool. We are working on a website for a complex workflow. This could take all of the business logic out of the web application (done via rails state plugins) and move it to the amazon service. Then just make semi-dumb forms to accomplish each state. I think that is the goal of this, you just have to code individual pages which coordinate with your database models and hit the API as the business layer.

Looks cool, but the link to the API reference is missing. I'd like to see some code examples and how you write non-java workflows.

From the product page: http://aws.amazon.com/documentation/swf/

It sounds scary to me to have something that core to your business be running on a rented platform.

Have you mapped out what services your business consumes and the relationships? e.g. Your bank accounts, payment systems, deliveries, suppliers. I think you'd be surprised.

Yes and no. There is a risk on building your business or app on somebody else's platform, but there is also risk in investing your time and resources on trying to recreate the same stuff (and be stuck maintaining it) rather than on your true core business. AWS is quickly becoming the standard runtime to build your application on top of (like win32 was in the 90s) and for a lot of scenarios the rewards on building on top of their platform are bigger than the potential for lock in.

Interesting. I just implemented a workflow thing for LiberWriter. It's a little unclear how the Amazon system might integrate with, say, an open ended comment system. Definitely something I'll be keeping an eye on though.

I didn't see it linked from the blog post, so here's the official SWF page: http://aws.amazon.com/swf/

Reply on news.ycombinator.com