tomasol

Really cool, the page is noticeably snappier. I'm wondering why they don't put the database on a network-attached block volume. DigitalOcean offers Volumes Block Storage, which is replicated across multiple hosts. Keeping SQLite there would allow the volume to be attached to a replacement Droplet rather than restoring from the latest nightly backup.

Subscribe to this issue: https://github.com/obeli-sk/obelisk/issues/65

sorry about that, I do love Beatles!

Check out https://obeli.sk/blog/comparing-dbos-part-1/

Hi, I wrote a comparison blog post between Obelisk and the Java version of DBOS a couple of months ago: https://obeli.sk/blog/comparing-dbos-part-1/

I believe the codegen must be separated from the runtime. Every time you ask AI for a new task, it must be deployed as a separate app with the least amount of privileges possible, potentially with manual approvals as the app is executing. So essentially you need a workflow engine.

fsync is the most expensive operation during a write. NORMAL mode means you don't care whether last ~100 ms of transactions before a process crash / VM restart are going to be persisted or not. My suggestion is either to use synchronous="full" or disable `synchronous_commit` on Postgres to avoid comparing apples to oranges.

Edit: Also, the example indicates financial transactions. Can you explain why you need serializability but not durability?

Author is setting PRAGMA synchronous="normal", meaning fsync is not issued as part of every write tx, but eventually. In order to make the comparison fair it should be set to "full".

Sorry for the late reply.

The actual order in which child workflows finish and their results hit the persistence layer is indeed nondeterministic in real-time execution. Trying to force deterministic completion order would likely be complex and defeat the purpose of parallelism, as you noted.

However, this external nondeterminism is outside the scope of the workflow execution's determinism required for replay.

When the workflow replays, it doesn't re-run the race. It consumes events from the log. The `-await-next` operation during replay simply reads the next recorded result, based on the fixed order. Since the log provides the same sequence of results every time, the workflow's internal logic proceeds identically, making the same decisions based on that recorded history.

Determinism is maintained within the replay context by reading the persisted, ordered outcomes of those nondeterministic races.

> Just create a race condition among a join set.

All responses and completed delays are stored in a table with an auto-incremented id, so the `-await-next` will always resolve to the same value.

As you mention, putting a persistent sleep and a child execution into the same join set is not yet implemented.

Thanks for bringing that up. Regarding the NaN canonicalization, there is a flag for it in wasmtime [1], I should probably make sure it is turned on.

Although I don't expect to be an issue practically speaking, Obelisk checks that the replay is deterministic and fails the workflow when an unexpected event is triggered. It should be also be possible to add an automatic replay of each finished execution to verify the determinism e.g. while testing.

[1] https://docs.rs/wasmtime/latest/wasmtime/struct.Config.html#...

Edit: Enabling the flags here: https://github.com/obeli-sk/obelisk/pull/67

I can do that. Please create an issue if you want to be notified about it.

The structured concurrency paradigm in workflows is stricter to what I'm reading here [1]. The whole execution model is different from the Task [2], as workflows are transparently unloaded and replayed.

Obelisk has a concept called join sets [3], where child executions are submitted and awaited. In the future I plan on adding cancellation and allow custom cleanup functions.

[1] https://github.com/WebAssembly/component-model/blob/main/des...

[2] https://github.com/WebAssembly/component-model/blob/main/des...

[3] https://obeli.sk/docs/latest/concepts/workflows/join-sets/

Indeed. I cannot compare the implementations as flawless is not open source. However on a high level they both share the same philosophy.

I believe the biggest difference is that Obelisk relies on the WASM Component Model:

Obelisk aims to avoid vendor lock-in. It is possible to write activities, workflows and webhooks with no obelisk SDK. Activities and webhooks are WASI 0.2 components that can be run on any compatible runtime like wasmtime e.g. for testing. This should also help with the adoption as any runtime will need a ton of integrations.

> I was referring to the workflows, that is, writing the workflows in managed languages, not the activities.

Ah understood. I have no plans supporting native workflow executors.

> Out of curiosity, are you working full-time on this?

Yes, currently I work on it full time.

> I'm working part-time on the same problem, looking to go full time soon, and it's interesting to see how the same ideas are popping up somewhat independently across different projects :)

Nice website! I also see the ideas of determinism, replayability etc more and more.

> let me know if you're interested in chatting!

Sure, my email is visible in the Git commit history.

> do I also need to compile activities to WASM

Yes, currently all activities must conform to the WASI 0.2 standard. This is the simplest for deployment, as you only need the obelisk executable, toml config file. The webhooks, workflows and activities pulled from a OCI registry on startup.

To support native code I plan to add external activities as well, with an interface similar to what Netflix Conductor uses for its workers.

> Finally, do you see the component's interface as the right layer for capturing IO?

An activity must encapsulate something much higher level than a single IO operation. So something like "Configure BGP on a router", "Start a VM" etc. It needs to be able to handle retries and thus be idempotent.

Regarding performance, a workflow execution can call 500-700 child executions serially, or around 1400 child executions concurrently per second.

Great questions. If you are fixing a bug in a workflow, which has running executions, there are two scenarios:

Either the fix does not break the determinism, meaning the the execution did not hit the fix yet. In this case the execution can be replayed and continue on the patched WASM component.

Otherwise, the execution replay causes "Non determinism detected" error. In this case you need to handle the situation manually. Since the execution log is in a sqlite file, you can select all execution affected by the bug and perform a custom cleanup. Also you can create a "forked" execution just by copying the execution log + child responses into a new execution, however there is no API for it yet.

> Can I update the workflow while it has running instances without interfering the running instances?

If you mean keep the in-progress executions on the old version of the code, you can do that by introducing a new version in the WIT file and/or change the new function name.

Yeah, looks like Golem went similar route - using WASM Component Model and wasmtime.

There is always this chicken and egg problem on a new platform, but I am hoping that LLMs can solve it partially - the activities are just HTTP clients with no complex logic.

Regarding the restrictions required for determinism, they only apply to workflows, not activities. Workflows should be describing just the business logic. All the complexities of retries, failure recovery, replay after server crash etc. are handled by the runtime. The WASM sandbox makes it impossible to introduce non-determinism - it would cause a compile error so no need for runtime checks.

Thanks for the kind words. In an ideal world I would like to offer a cloud version that would be monetized. There are a few examples on how to do HTTP requests, I have a demo repository [1] with GraphQL and regular JSON-over-HTTP activities. I do agree that the ecosystem is not mature yet, but I was able to generate HTTP activities using LLM on a single shot.

1: https://github.com/obeli-sk/demo-stargazers

The biggest motivator for me is that WASM sandbox provides true deterministic execution. Contrary to engines like temporal, using hashmaps is 100% deterministic here. Attempting to spawn a thread is a compile error. It also performs well - the bottleneck is in the write throughput of sqlite. Last but not least - all the interfaces between workflows and activities are type safe, described in a WIT schema.