Tools for converting Python code to AWS Step Function JSON

Great to see open source tooling around Step Functions. Personally I'm still holding out for someone taking the States language spec (https://states-language.net/spec.html) and turning out higher-level tooling for writing/inspecting/manipulating state machines.

Agree w/ bpicolo (hi Ben!) that the tooling still can come a long way for those who can't use Lambdas directly. This code is interesting, but it misses my major use case for Step Functions -- running a workflow across several services transparently.

If I had to guess the future, I suspect that tools like Step Functions will become more and more crucial as we realize that in a FaaS world the job of coordination is complex and essential.

Interesting video from Tim Bray showing off the variety of Step Function uses in the wild https://www.youtube.com/watch?v=sMaqd5J69Ns. Lots of Lambda, but not solely!

Step Functions have great potential for orchestrating complex executions of Lambdas or even fleets of them. My co-workers and I designed a threadpool pattern with step functions that we use for orchestrating complex long-running tasks: https://github.com/kislyuk/domovoi/blob/master/domovoi/examp...

However, there are currently some big issues with step functions: compared to Lambdas, they are not very scalable, not very economical, and have low I/O size limits (because all I/O gets interpreted by the executor, so they limit the state size to 32KB). Running Lambdas, we're used to be able to burst to many thousands of concurrent executions. The Step Functions burst start limit is 500, with just 25 per second after that (and 400 for state transitions). This can be seriously limiting if you need to process more activities in your system.

Step functions have a lot of good use cases - strong guarantees, concurrent, distributed, good at recovery in all the right ways. It's a terrific queueing layer abstraction for complex business processes. I suspect there are a lot of businesses out there with adhoc state machine definitions in queue workers that would benefit from formalization.

I had a closed-source proof of concept for a little framework for building these in Python while at a former employer that worked out pretty neat. The gist was arranging / building them more in pure code and automatically generating/deploying lambda tasks and state machine.

    @task(name='foo')
    def some_lambda(input):
        ....
    
    state_machine = StateMachine(
      {...define_some_steps}
    )

then it was able to take advantage of Zappa to immutable deploy steps as lambdas and build new state machine definitions with a little bit of python magic. Was pretty nifty and not particularly tricky to make it work.

Amazon would do really well releasing a maintained open-source lib / docker container that would be able to emulate state machines locally. The definition language is pretty simple, but it's something that you definitely want to be able to test end-to-end. I at one point did a little PoC for that bit too (https://github.com/bpicolo/local-step-functions).

The one big miss for me in step functions right now is that Tasks can't just POST to an HTTP endpoint for request/response to take advantage of existing infrastructure/http services easily. That'd be a real killer feature. They're currently distributed in nature requiring lambdas or polling daemons, which is powerful but also requires a lot of operational / configuration / deployment overhead where a simpler model would suffice for many use cases - seems like there's no reason you shouldn't be able to just deploy a monolith and get a lot of use out of step functions in that regard. There are ways you could sort of shim it with lambdas but the overhead is pretty high to do that (you'd need to carry / edit routes in the task data which would ruin the declarative nature of task definitions)

This seems similar to Apache Airflow:

https://github.com/apache/incubator-airflow/blob/master/READ...