The Workflow Pattern

> I always thought it was a bit silly that the async mechanism in modern languages is so myopic and single-purpose. These kinds of high level language transformations ought to be extensible and pluggable so that we could write workflows in a proper programming language and have it look like normal code except for the occasional “await” keyword.

This is how Temporal works. For example, in Python the async event loop is replaced by a durable event loop [0], the JS promises become durable promises, .NET tasks become durable via a custom scheduler, etc. Granted it doesn't serialize the stack, it uses event sourcing almost exactly like the article describes, and therefore requires deterministic code for replaying. From the dev POV, it looks like any code can just be frozen in the middle of a function and magically resumed elsewhere.

0 - https://temporal.io/blog/durable-distributed-asyncio-event-l...

(disclaimer, I work at Temporal and have written some of these distributed coroutine impls)

Workflows engines are not just about deferring a http call or a specific language.

They are a generic way to run some background process that doesn't need too much human interaction and can be coded in any language. If you need to orchestrate tens of thousands of tasks, you will beg for a DSL and some kind of interface to monitor things.

The missing feature you're describing is monads. They can essentially be a first-class abstraction for computation (among other things). They can represent synchronous computations, computations that can be suspended/resumed, computations that might fail with a particular type of exception, or anything in between.

and there is always the Long-Running-Transactions. Where the workflow/state-machine can take days or weeks to finish. Somehow, we all still live with some hope that certain things will happen real fast so they can live in threads/processes/memory (and have no persistent backing, except maybe retry whole thing), while those that can take "longer" (how long is "longer"?) should be completely different beasts. While from business-level view, there is no much difference - but they are programmed in completely different way.

For example, in some businesses, payment is like another communication. Not that sending a single message across is very simple thing, but definitely a lot less complicated than a payment process.

Maybe the split, should it be internaly-language-handled - and forgotten if power goes off - or externalized and persisted, should depend on where there is certain boundary being crossed? Defining "boundary" per project..

Sounds like Azure Durable Functions: https://learn.microsoft.com/en-us/azure/azure-functions/dura...

Rather than focus on the DSL/code-writing aspect of workflow pattern tools, I feel the crucial difference is that most workflow systems are intended for a broad audience of users, many of whom are not devs.

I have similar doubts about how important the difference is between programming as devs do it and these tools. But to me, it's less a language concern and more a question of how tasks/jobs/processes/whatever-you-call-them are managed, if they're managed at all. In programming languages today, devs spawn countless async sub-processes (promises, async tasks, whatever you call them) with abandon. And they're implicit and invisible in almost all systems. An operator or user isn't there introspecting what jobs are outstanding. Or managing the workflow of these subprocesses.

These workflow tools are just a tiny leap over what programmers do: they make computing real. They reify computing into a managed element, make work being done a kind of data that is tracked through the system. Where-as in most programming, we have lots of tools for authoring and finagling subprocesses, but we are so very lacking in tools to manage & expose & let the user control what's happening in the process. The processs is mostly-closed, except what is afforded, and what is afforded is largely custom-craft APIs (with folks like OTel starting to define some more common ways for processes to at least explain themselves, if not manage & interconnect with other processes).

I don't think the languages have to change. I think the languages can stay right where they are, for the most part. But we can and should be building better runtimes, that let us interact with the objects & subprocesses, that let the app be more commanded and turned into an task runner rather than also the command-pallet system, kind of has to happen. It's not super related, but there was a post on Audacity in the browser, where @solardev suggested "headless" WASM apps and then separate UIs, and I can easily see that idea applying here: a workflow pattern tool being the UI, that orchestrates/drives WASM apps (which are practically just libraries).

The async/await stuff in JavaScript, at least, is supported by the “generators” feature, which does let you effectively “pause” execution of a function, retaining its state. You can then yield to that state to continue executing

My company has 200+ DAGs on Airflow, so that ship has sailed for me.

That said, Conductor looks really cool, thanks for sharing.

I recently created some Netflix Conductor workflows with Python workers and found it pretty good. Using a fork/join to kick off sub workflows was pretty easy. It definitely shows runtime variable data much better than Camunda.

this is exactly how I go about re-architecting overgrown workflows. reduce the workflow to a state machine. It takes forever to untangle what they are wanting to do and what they are doing, but the result is always a more robust solution.

the other benefit of a statemachine is the ability to accurately determine what parts can be collapsed into subworkflows which allows for reuse, replacement, or general modification

At a prior company we just wrote our own custom workflow engine which was basically a DAG processor where each node called some code. Of course, the code called could modify the DAG in place and accomplish exactly what you're describing. It worked quite well but we didn't polish it as much as we could have. I've been wanting to do a clean-room reimplementation as of late just because I miss certain things about it.

Not sure what you've tried, but Temporal (which I think is the most mature and well-designed open source workflow engine available right now) supports using the result of a "subprocess" to decide the next steps. The only requirement is that your logic is consistent across workflow runs.

That sounds very similar to AWS Step Functions to me. You define a workflow graph in JSON, then run the workflow and hand out subprocesses to lambdas etc.

The couple workflow engines I've used supports that. It sounds crazy to not support something that obvious

Maybe before, but i feel that is not a problem anymore. Windmill.dev solves this with easee. Very good UI on top of the workflow engine.

Why would you want to rewrite the code of the workflow during execution, vs defining that logic upfront?

Interesting; for me NiFi is tied with Airflow as my most-hated software I've had to use professionally.

This was a few years ago but NiFi only supported editing the DAG from a GUI; you could store the resulting XML in git, but no human could read or edit it. Python steps were stuck on Python 2.6 or 2.7. Hard to debug pileups. Is it better now?