Hacker News Reader

> I always thought it was a bit silly that the async mechanism in modern languages is so myopic and single-purpose. These kinds of high level language transformations ought to be extensible and pluggable so that we could write workflows in a proper programming language and have it look like normal code except for the occasional “await” keyword.

This is how Temporal works. For example, in Python the async event loop is replaced by a durable event loop [0], the JS promises become durable promises, .NET tasks become durable via a custom scheduler, etc. Granted it doesn't serialize the stack, it uses event sourcing almost exactly like the article describes, and therefore requires deterministic code for replaying. From the dev POV, it looks like any code can just be frozen in the middle of a function and magically resumed elsewhere.

0 - https://temporal.io/blog/durable-distributed-asyncio-event-l...

(disclaimer, I work at Temporal and have written some of these distributed coroutine impls)

Workflows engines are not just about deferring a http call or a specific language.

They are a generic way to run some background process that doesn't need too much human interaction and can be coded in any language. If you need to orchestrate tens of thousands of tasks, you will beg for a DSL and some kind of interface to monitor things.

I feel like awaiting some human interaction is one of my most common requests for workflows. await some approval or some manual task that can't be automated, etc.

honestly, when working with some new archaic system, the ability to insert manual tasks is invaluable for an early deployment

The missing feature you're describing is monads. They can essentially be a first-class abstraction for computation (among other things). They can represent synchronous computations, computations that can be suspended/resumed, computations that might fail with a particular type of exception, or anything in between.

This is one way to structure it but not the only. Scheme's call-with-current-continuation comes to mind here.

One really nifty use case of continuations is the Seaside Smalltalk web framework [1]. It implements stateful request/response conversations with Smalltalk's ability to capture and serialize continuations.

Continuation-based web frameworks are really nifty, but have potential scaling bottlenecks. IIRC HN originally used some type of continuation web framework with Arc, but ran into scaling and caching headaches. Dunno how it works these days.

[1] https://en.wikipedia.org/wiki/Seaside_(software)

I didn't know it can serialize continuation! I always thought that with seaside you have to run a single server, and any reboot would just kill the state.

I didn't think callCC can do multiple resumptions? DDBinding certainly can, and can do partial environment closures so you can migrate a process to a different machine as well: https://okmij.org/ftp/papers/DDBinding.pdf

The above is succinctly implemented in wat: https://github.com/manuel/wat-js (also see forks for more documented versions).

and there is always the Long-Running-Transactions. Where the workflow/state-machine can take days or weeks to finish. Somehow, we all still live with some hope that certain things will happen real fast so they can live in threads/processes/memory (and have no persistent backing, except maybe retry whole thing), while those that can take "longer" (how long is "longer"?) should be completely different beasts. While from business-level view, there is no much difference - but they are programmed in completely different way.

For example, in some businesses, payment is like another communication. Not that sending a single message across is very simple thing, but definitely a lot less complicated than a payment process.

Maybe the split, should it be internaly-language-handled - and forgotten if power goes off - or externalized and persisted, should depend on where there is certain boundary being crossed? Defining "boundary" per project..

Sounds like Azure Durable Functions: https://learn.microsoft.com/en-us/azure/azure-functions/dura...

Rather than focus on the DSL/code-writing aspect of workflow pattern tools, I feel the crucial difference is that most workflow systems are intended for a broad audience of users, many of whom are not devs.

I have similar doubts about how important the difference is between programming as devs do it and these tools. But to me, it's less a language concern and more a question of how tasks/jobs/processes/whatever-you-call-them are managed, if they're managed at all. In programming languages today, devs spawn countless async sub-processes (promises, async tasks, whatever you call them) with abandon. And they're implicit and invisible in almost all systems. An operator or user isn't there introspecting what jobs are outstanding. Or managing the workflow of these subprocesses.

These workflow tools are just a tiny leap over what programmers do: they make computing real. They reify computing into a managed element, make work being done a kind of data that is tracked through the system. Where-as in most programming, we have lots of tools for authoring and finagling subprocesses, but we are so very lacking in tools to manage & expose & let the user control what's happening in the process. The processs is mostly-closed, except what is afforded, and what is afforded is largely custom-craft APIs (with folks like OTel starting to define some more common ways for processes to at least explain themselves, if not manage & interconnect with other processes).

I don't think the languages have to change. I think the languages can stay right where they are, for the most part. But we can and should be building better runtimes, that let us interact with the objects & subprocesses, that let the app be more commanded and turned into an task runner rather than also the command-pallet system, kind of has to happen. It's not super related, but there was a post on Audacity in the browser, where @solardev suggested "headless" WASM apps and then separate UIs, and I can easily see that idea applying here: a workflow pattern tool being the UI, that orchestrates/drives WASM apps (which are practically just libraries).

The async/await stuff in JavaScript, at least, is supported by the “generators” feature, which does let you effectively “pause” execution of a function, retaining its state. You can then yield to that state to continue executing

Sure, but would you still be able to resume that function 2 weeks later, after 5 process restarts due to code deploys and on a different server?

CRIU can apparently do that with whole processes: https://criu.org/Main_Page

Found a link to it from telefork: https://thume.ca/2020/04/18/telefork-forking-a-process-onto-...