AWS Step Functions – Build Distributed Applications Using Visual Workflows

The Flow Framework is an abomination to engineering, and should die a pathetic death. Liberal annotations, codegen, aspect weaving and bytecode manip, requires a special JUnit test runner--it embodies all the aspects of the Java culture that I despise.

I rather like the idea of SWF, but its API is too low-level, it has poor tooling, and the workflow/activity versioning is awkward and makes the systems too rigid. I am glad that AWS is moving to data-driven workflow definitions, that means they can build much better operational tooling (like the GUI) to manage and understand workflow executions.

Well similar to SWF it looks like they provide the infrastructure for tracking the state changes and the input/output data at each step so you can:

  (1) Quickly get a visual overview of all your executions

  (2) See  how many workflows failed 

  (3) Drill down and see which specific step failed and the data that might have triggered the failure 

  (4) re-run specific workflows

It provides built in error handling, retries/backoff, and parallelization. These are all things you could code yourself, but at some point you would have built a workflow system. Not to mention that the infrastructure of the workflow system itself is managed for you, so you don't have to worry about that going down. You can write workflows that execute over days, weeks or months and not worry about what happens if your server goes down.

It would also come in handy for mutli-language workflows. If your were interacting with a variety of third party APIs that had SDKs in different languages, you could quickly string together a Python Lambda, Java Lambda and some .NET code that used the long polling API.

It would be nice if it integrated seamlessly with Cloudwatch and X-Ray to provide a unified view of logging and tracing right from the same interface.

I also expect a more seamless integration with AWS Batch at some point. Could be useful for chaining together multiple independently maintained batch processes (video conversion, resizing, publish to youtube).

I personally use SWF for a 10 step e-commerce workflow that calls multiple API. It works great, but I could have developed it much more quickly using Step Functions if it were available at the time.

One major advantage over its predecessor is the first image: your workflow is defined as data instead of code, so you get a nice visualization of your workflow state. Try reading SWF execution histories for comparison.

Are you asking why you wouldn't just write your own coordination code and not use a coordination service at all? The advantage is scaling and robustness. These workflows can run for days/weeks/months, and each step in the workflow can run on different hosts that are optimized for that step. And workflow state is handled by AWS, and so is basic workflow logic (retries, timeouts, event logging, etc.). Unlike Oozie, Luigi, etc., your workflow state is not coupled to any of your infrastructure, so it can survive all kinds of crazy situations (fleets dropping offline, replacing clusters, etc.).

Another example just occurred to me. Say for example that I need to download a CSV from a third-party everyday, but their server is not reliable, so sometimes it times out, or sometimes the file is not uploaded when it should be.

With Step Functions it is now trivial for me to write a simple Python lambda to do the HTTP request and then have the state machine deal with retrying with an exponential backoff (30min, 2hr, 8hr) for as long as I want.

If you wrote that by hand you would either need a long running process or you would need to manage scheduling and state management yourself.

I was actually planning on doing something like this with lambda, but I was basically just going hack something together to fire the Lambda multiple times per day using CloudWatch Events and then use DynamoDB to keep track of whether or not I had already succeeded. This is much more elegant and robust and you get a visual overview and execution history for free.

A primitive version ... but it does look a LOT like it

I didn't know what that was, and went to HP's website to look it up.

> "IT process automation and run book software that improves service quality and customer satisfaction, and lowers costs by eliminating latency between silos, increasing first time right rate, enforcing standards, and delivering reports for ROI, audits and more."

Ugh, seriously HP? What a load of BS. What the hell does it do?

Compare that to the AWS description of Step Functions:

> "AWS Step Functions makes it easy to coordinate the components of distributed applications and microservices using visual workflows. Building applications from individual components that each perform a discrete function lets you scale and change applications quickly."

Reminds me a lot of BPEL back in the day. Great now I'm having flashbacks

> Those graphical programming interface all end up miserably.

This is factually not true. There are a number of fields where graphical programming interfaces have a long history of success.

This isn't really a graphical programming interface. The stuff inside of the blocks is code that's deployed separately and does a thing. There's a graphical thingy that allows you to describe state transitions between those blocks of code but you aren't really programming.

False. By a long shot.

Just look at LabVIEW & Operations Orchestration.