Solving durable execution's immutability problem
restate.dev
restate.dev
Ironically, I'm still largely about idempotent calls that should be able to trigger something. And those idempotent identifiers can be stored such that you can replay or serve up the relevant data, as needed. But going all in and forcing a long compute chain that cannot be changed is something you do in places you have to do it. No reason to choose to do it everywhere. And certainly don't build larger and larger compute chains between durable writes to the backing data store.
I don't see any reason whatsoever that a temporal workflow composed of activities would be subject to the things you've described. And I don't even think it's that impressive of an implementation of the idea.
My complaint was less that it isn't a compelling idea. The problem seems to be that people want to expand the one system they are working on to be a fully comprehensive system. I remember one team tried to get their entire state reflected in a Step Function state machine that could run for as long as necessary. Idea literally being a state machine that would last years or so. They compromised for one that would only last a week or so. And then faced the consequences rather quickly.
And note, given enough time, I think this can work. On a deadline, though, I've seen it fail too many times for it to be my goto solution. All the time by people that are objectively smarter than I am.
Very fair, and point totally understood.
(obviously the rules are different for KV/object/blob-stores and don't apply here).
-----
That said, if you need a stopgap solution for a problem like that, then take advantage of existing compatibility-shim features in a decent RDBMS: things like VIEWs and SYNONOYMs exist for this reason (and yes, you can INSERT INTO-a-VIEW).
-----
Another option is to constrain your DB designs such that all DDL/schema changes are always backwards-compatible (i.e. disallow dropping anything, disallow alter-column except to widen an existing column (e.g. int-to-bigint is okay but not string-to-UUID), all domain-model attributes must be initially modelled with a m:m multiplicity whereever possible, etc etc) - and you can enforce these rules using a DDL trigger too - and any changes made by a workflow's DML can be repaired by a background job.
----
(Just throwing some ideas around)
One effective pattern I’ve seen in large DB deployments is to separate the write schema from the read schema. That is, treat what is allowed to be written into the DB separately from the shapes of any views that exist. The views themselves are a tightly coupled client of the DB log - by constraining writers, you can migrate/rebuild views, then point services to read from the migrated views, and retire old views.
This allows you to keep accepting writes - you never have to shut down the write path. If you’re introducing new shapes or the DB, you’d prepare a new view, the “widen” the write schema, and begin accepting writes in the new shape, and only then re-point clients to read from the new view.
To drop elements of your read schema, you do the dance in reverse. First, constrain your writes. Then, build new views that don’t require the elements you removed. Gradually, update application code to work on the new, reduced views. When you’re done reading from the original views, you can drop them.
This is inherently much less efficient than online or offline DB migrations. But it’s a sane strategy for wrangling very large systems with very low risk.
Versioned workflows are in practice distributed entities that interact with their peers, themselves defined by versioned interfaces. By tracking the versions of handlers touched by any given execution, we can imagine a similar experience for deployed code - including garbage-collecting unused versions.
For a post about "solving" durable execution I would expect both a scale-to-zero way to keep older versions around indeterminately - I guess the Lambda based approach does qualify - and a safe and controlled way to upgrade task versions iff the execution history is compatible.
If you replay history up to the newest call and all calls are identical, that specific execution instance is compatible with the new code and can be upgraded. If not it should be rolled back, and you can either deploy a fixed version of the code with backwards compatibility, or delete executions that can not be upgraded.
Backwards compatible code can be written as
if (workflowVersion() >= FIX_VERSION) new_way() else old_way()
originalWorkflowVersion() is constant, e.g. all executions that started before NEW_TAX_RULE will keep using the old tax rules for all calculations.
Would be keen for a python implementation