KraftyOne

As the post says, the exactly-once guarantee is ONLY for steps performing database operations. For those, you actually can get an exactly-once guarantee by running the database operations in the same Postgres transaction as your durable checkpoint. That's a pretty cool benefit of building workflows on Postgres! Of course, if there are side effects outside the database, those happen at-least-once.

(DBOS co-founder here) From a DBOS perspective, the biggest differences are that DBOS runs in-process instead of on an external server, and DBOS lets you write worklflows as code instead of explicit DAGs. I'm less familiar with Hatchet, but here's a blog post comparing DBOS with Temporal, which also uses external orchestration for durable execution: https://www.dbos.dev/blog/durable-execution-coding-compariso...

(DBOS co-founder here) DBOS embeds durable execution into your app as a library backed by Postgres, whereas Restate provides durable execution as a service.

In my opinion, this makes DBOS more lightweight and easier to integrate into an existing application. To use DBOS, you just install the library and annotate workflows and steps in your program. DBOS will checkpoint your workflows and steps in Postgres to make them durable and recover them from failures, but will otherwise leave your application alone. By contrast, to use Restate, you need to split out your durable code into a separate worker service and use the Restate server to dispatch events to it. You're essentially outsourcing control flow and event processing to the Restate server., which will require some rearchitecting.

Here's a blog post with more detail comparing DBOS and Temporal, whose model is similar to (but not the same as!) Restate: https://www.dbos.dev/blog/durable-execution-coding-compariso...

Tables getting too big hasn't been a concern in practice because information on completed workflows can easily be GC'ed.

Yeah, this definitely requires splitting state-impactful activity into its own function. That's good practice anyways, though I understand it might be a pain in large codebases. Regional tags are definitely an interesting alternative!

For the UI example, I don't think you'd use durable execution for most of the UI--it's just not needed. But maybe there's one button that launches a complex asynchronous background task, and you'd use durable execution for that (with careful workflow ID management to ensure idempotency and allow you to retrieve the status of the background task).

Agreed, if you can do something completely synchronously while responding to an HTTP request, you should.

But often you can't! Then, durable execution helps you manage the complexity of async processing.

DBOS solves this problem by storing state in Postgres, which is really good at coordinating multiple copies of the same service. Essentially, Postgres does the hard parts of external orchestration, letting you work with a simple library abstraction.

This is a great answer, and yes, those are critical aspects of durable execution. Maybe I should write a follow-on post that goes into more detail...

Thanks! DBOS is simpler not because it ignores complexity, but because it uses Postgres to deal with complexity. And Postgres is a very powerful tool for building reliable systems!

Yes, that's totally fair. Usually, a step is a meaningful unit of work, such as a API call that performs an external state modification. Because each step is a fair chunk of work, and the overhead is just one write per step, this scales well in practice--as well as Postgres scales, up to 10K+ operations/second.

That's exactly what this model is! The @Step decorator is for external state modifications. Then @Workflows orchestrate steps. The example shows the simplest possible external state modification--a print to the terminal.

Steps can be tried multiple times (if a failure happens mid-step) but never re-execute once complete. Since idempotency can't be added externally, that's the strongest possible guarantee any orchestration system can give you (and if your step is performing an idempotent operation, which is the safest thing, you can use the workflow ID as an idempotency key). More details in the docs: https://docs.dbos.dev/python/tutorials/workflow-tutorial#rel...

That's really interesting! It does seem that this is identically semantically to the library approach (as the logic your interpreter adds around steps could also be added by decorators) but is completely automatic. Which is great if the interpreter always does the right thing, but problematic/overly magical if the interpreter doesn't. For example, if your problem domain has two blocking operations that really form one single step and should be retried together, a library approach lets you express that but an interpreted approach might get it wrong.

Generally we recommend against retroaction--the assumed model is that every workflow finishes on the code version it started. This is managed automatically in our hosted version (DBOS Cloud) and there's an API for self-hosting: https://docs.dbos.dev/typescript/tutorials/development/self-...

That said, we know sometimes you have to do surgery on a long-running workflow, and we're looking at adding better tooling for it. It's completely doable because all the state is stored in Postgres tables (https://docs.dbos.dev/explanations/system-tables).

Great question! A workflow should be deterministic: if called multiple times with the same inputs, it should invoke the same steps with the same inputs in the same order. But steps don't have be deterministic, they can invoke LLMs, third party APIs, or any other operation. Docs page on determinism: https://docs.dbos.dev/typescript/tutorials/workflow-tutorial...

That's a really good question! Because DBOS is backed by Postgres, it scales as well as Postgres does, so 10K+ steps per second with a large database server. That's good for most workloads. Past that, you can split your workload into multiple services or shard it. Past that, you've probably outscaled any Postgres-based solution (very few services need this scale).

The big advantages of using Postgres are:

1. Simpler architecturally, as there are no external dependencies.

2. You have complete control over your execution state, as it's all on tables on your Postgres server (docs for those tables: https://docs.dbos.dev/explanations/system-tables#system-tabl...)

It's not recommended--the assumed model is that every workflow finishes on the code version it started. This is managed automatically in our hosted version (DBOS Cloud) and there's an API for self-hosting: https://docs.dbos.dev/typescript/tutorials/development/self-...

That said, we know sometimes you have to do surgery on a long-running workflow, and we're looking at adding better tooling for it. It's completely doable because all the state is stored in Postgres tables (https://docs.dbos.dev/explanations/system-tables).

Got it! What specifically are you looking for? If you launch multiple DBOS instances connected to the same Postgres database, they'll automatically form a distributed task queue, dividing new work as it arrives on the queue. If you're looking for a lightweight deployment environment, we also have a hosted solution (DBOS Cloud).

Yes, if your workflow interacts with multiple external systems and you need it to fully back out and clean up after itself after a step fails, you'll need backup steps--this is basically a saga pattern.

Where DBOS helps is in ensuring the entire workflow, including all backup steps, always run. So if your service is interrupted and that causes the Stripe call to fail, upon restart your program will automatically retry the Stripe call and if that doesn't work, back out and run the step that closes out the failed purchase.

That work is done by the decorators! They wrap around your functions and store the execution state of your workflows in Postgres, specifically:

- Which workflows are executing

- What their inputs were

- Which steps have completed

- What their outputs were

Here's a reference for the Postgres tables DBOS uses to manage that state: https://docs.dbos.dev/explanations/system-tables

That's definitely worth considering! The core algorithms can work with any data store. That said, we're focused on Postgres right now because of its incredible support and popularity.

Yeah, the arguments and return values of steps have to be serializable to JSON.

For versioning, each workflow is tagged with the code version that ran it, and we recommend recovering workflows on an executor running the same code version as what the workflow started on. Docs for self hosting: https://docs.dbos.dev/typescript/tutorials/development/self-.... In our hosted service (DBOS Cloud) this is all done automatically.

The state can be stored in any Postgres instance, either locally or in any cloud.

For code, here's the bare minimum code example for a workflow:

  class Example {
    @DBOS.step()
    static async step_one() {
      ...
    }

    @DBOS.step()
    static async step_two() {
      ...
    }

    @DBOS.workflow()
    static async workflow() {
      await Example.step_one()
      await Example.step_two()
    }
  }

The steps can be any TypeScript function.

Then we have a bunch more examples in our docs: https://docs.dbos.dev/.

Or if you want to try it yourself download a template:

    npx @dbos-inc/create

For separation of concerns/isolation, DBOS does all its bookkeeping in a separate Postgres database from your app database (documented here: https://docs.dbos.dev/explanations/system-tables).

And yes, you can absolutely run this locally as a library! The bottom half of the quickstart covers how to do it: https://docs.dbos.dev/explanations/system-tables Just use a local (or Docker) Postgres.

Building a complex stateful system will always be hard, but workflows as an abstraction have two big benefits:

1. Automatic handling of any transient failures or service interruptions/crashes/restarts. Transient failures in steps are automatically retried, and service interruptions are automatically recovered from. Even if you're doing your own bookkeeping, doing _recovery_ from that bookkeeping isn't easy, and workflows do it automatically.

2. Built-in observability. Workflows naturally support built-in observability and tooling that isn't easy to build yourself. DBOS integrates with OpenTelemetry, automatically generating complete traces of your workflows and giving you a dashboard to view them from (and because it's all OTel, you can also feed the traces into your existing obs infrastructure). So it's easier to spot a step getting stuck or failing unexpectedly.

Another advantage of DBOS specifically, versus other workflow engines, is that all its bookkeeping is in Postgres and well-documented (https://docs.dbos.dev/explanations/system-tables), so you have full and complete control over your workflows if you need it.

Yes, that's why the @DBOS.Transaction is just for database operations, which (hopefully!) you've already optimized to not take unreasonable time. These are guaranteed to execute exactly-once. For other, potentially longer-running operations, use @DBOS.step, which is at-least-once. Documentation:

Transactions: https://docs.dbos.dev/python/tutorials/transaction-tutorial

Steps: https://docs.dbos.dev/python/tutorials/step-tutorial

Got it! So you can make the entire HTTP endpoint a DBOS workflow that performs the synchronous work then launches a background task. Something like this:

    @app.get("/endpoint")
    @DBOS.workflow()
    def http_workflow():
        synchronous_task() # Run the synchronous task
        DBOS.start_workflow(background_task) # Start the background task asynchronously

This is atomic in the sense that if the synchronous task runs, the background task will always also run.

I haven't used Conductor, but quickly looking at the README, Conductor lets you define JSON workflows to orchestrate existing microservices. By contrast, DBOS helps you build highly reliable applications--it runs as a library inside your program and helps you build durable workflows that run alongside your code and are written in the same language.

Compared to Celery, DBOS provides a similar queuing abstraction (Docs: https://docs.dbos.dev/python/tutorials/queue-tutorial) DBOS tries to spread out queued tasks among all workers (on DBOS Cloud, the workers autoscale with load), but there isn't yet support for running specific tasks on specific workers. Would love to learn more about that use case!

Compared to Dagster (or Prefect or Airflow), exactly like you said, a DBOS workflow is basically a more flexible and lightweight DAG. The visualization piece is something we're actively developing leveraging OpenTelemetry--look for some cool new viz features by the end of the month! I'm interested in the "retry step 2 only" or "retry from step 2" use cases--would love to learn more about them--we don't currently support them but easily could (because it's all just Postgres tables under the hood). If you're building in this space please reach out, would love to chat!

Got it! I'm not sure if that was what OP was asking, but it's a really interesting question.

We don't currently support launching a workflow atomically from within an existing database transaction. I'd love to learn about the use case for that!

We do support calling a database transaction as a workflow step, which executes entirely atomically and exactly-once: https://docs.dbos.dev/python/tutorials/transaction-tutorial

Yes exactly! That's how Postgres synchronous replication works. If you turn on synchronous replication, then you won't lose data unless all your replicas are wiped out. The question the original poster was asking was what guarantees can be provided if you DON'T use synchronous replication--and the answer is that there's no such thing as a free lunch.