hmaxdml

I thought that fell under BYOD (device) :)

What's reasonable? DBOS has users running queues at millions of tasks per hour.

For fair queuing you can have partitioned queues where only active partitions consume resources

Not saying this in a mean way:

It seems to me this can be boiled down to "using things without understanding how they work doesn't scale". Yes, vanilla listen/notify doesn't scale. But the OP actually figured out how to make it scale. So your engineers don't have to.

As the community builds, over time, distributed systems, it also understands which brick can do what, and it turns out that starting with less bricks and adding some when you actually need them makes for healthier systems.

The database is exactly the hardcore piece of engineering that's been designed to scale and be fault tolerant for decades

Because you likely already have a database and likely don't need to bring on an entire new distributed system to orchestrate your workflows.

A PG-backed queue is in code right after being in PG, and the beauty of a neat durable queue framework is in exposing it conveniently and efficiently.

Postgres does scale pretty well: https://www.dbos.dev/blog/benchmarking-workflow-execution-sc...

Tens of thousands of workflows per second

That's why their entire business model -- like Astronomer's -- is geared toward cloud hosting. The architecture is so complex it takes a full time team to run it.

Have you looked into DBOS? Same thesis: durable and reliable workflows are hard to manage -- it just doesn't have to be as hard as Temporal makes it be :)

DBOS python supports SQLite. Go is supporting it next release

I've talked to dozens of engineers who built their home grown "durable" stack. Most of them eventually moved on to buying vs building, when their system actually scaled. It's just not a side-hustle to build a foundational reliability layer.

Yeah, we've observed that too: people start implementing their own retry logic, idempotency, etc. But then they grow a hard to maintain, complex stack that's not their core business logic. There's a reason why there is a dedicated team building DBOS, every day. Because it's not that easy to build a solid durable workflows engine on Postgres.

Listen/notify is poised to become much better in PG 18 and 19

As you said, the example is simple and it might not be obvious to people without prod experience what the problems can be. Postgres can give you all the primitives you need to solve this at the application layer. Durable workflows on Postgres is an effective way to access these primitives.

We've found that durable workflows is a much needed primitive for agents control flow. They give a structure for deterministic replays, observability, and, of course, fault tolerance, that operators need to make the agent loop reliable.

Automatic crash detection for your process is built-in our Conductor offering. The library has a default recovery mode when used standalone.

What do you find strange with workflow versioning? Would love to consider improving the semantics. In fact, we started doing it: https://github.com/dbos-inc/dbos-transact-py/pull/598

I notice you didn't provide any specific comparison alongside that comment, which makes me feel frustrated because I think the Temporal workflow SDK is very different. Architecturally, Temporal and DBOS are at two opposites of the durable execution spectrum. I'd love to understand what makes you think this work is a mere copy and paste. Would you be willing to share some more with me?

Thanks for the comment (author here). I wanted this post to focus on the Golang specific implementation, not dwell on the durable execution ecosystem at large.

With respect to context, I don't know that anyone invented "having their own context". Go interface are extendable and pretty much every major framework I know of implement their own context.

Would love to learn more about the gaps that offset you. We're constantly improving here ;)

Conductor is about enterprise features like automatic workflow recovery, alerting, or RBAC. The GUI is a nice to have -- but all your workflow data are in Postgres. You can access it very easily.

Durable execution has already been mentioned as the existing solution for this problem, but I would like to call out a specific pattern that DE makes obsolete: the outbox pattern. Imagine just being able to do do

send a() send b()

And know both will be sent at least once, without having to introduce an outbox and re-architect your code to use a message relay. We can nitpick the details, but being able to "just write normal code" and get strong guarantees is, imo, real progress.

These are all important concerns, but I'd go for an off the shelf library that does it for me (disclaimer I work at https://github.com/dbos-inc)

The hype is because DE is such an dev exp improvement over building your own queue. Good DE frameworks come with workflows, pub/sub, notifications, distributed queues with tons of flow control options, etc.

one way is to follow https://www.linkedin.com/company/dbos-inc for updates

Good call. We'll see how to integrate it in our docs better.

The cost of DBOS durable execution is 1 write per step (checkpoint the outcome) and 2 additional writes per workflows (upsert the workflow status, checkpoint the outcome). The write size is the size of your workflows/steps output.

Postgres can support several thousands writes per seconds (influenced by the write size, ofc): DBOS can thus support several thousands of workflows/steps per second.

Postgres scales remarkably well. In fact, most org will never out scale a single, vertically scaled Postgres instance. There's a very good write up by Figma telling how they scaled Postgres horizontally: https://www.figma.com/blog/how-figmas-databases-team-lived-t...

DBOS stores all the workflow metadata in postgres, which is readily queryable for observability. We've recently seen a user setup an entire Grafana dashboard to observe their numerous workflows.

A postgres server can host many databases, and multiple applications can use the same server. The same dashboard can be used to monitor them all.

With respect to recovery: A new Transact process will run a round of recovery at startup. Transact also exposes an admin server with a recovery endpoint.

For more elaborate scenarios, we have control plane options commercially available.

We decided to use Postgres because of the relational semantics, the ease of integration with user applications, and it's remarkable popularity

It is an evolution. The DBOS workflow orchestrator places a DB at the center of your application to handle most of the complicated state management problems.

Thanks for posting! I am one of the author, happy to answer any question!

:wave: Hey there, I'm working on the Go library and just wanted to confirm your suspicion:

"since Golang doesn't have decorators in the same way Python does, we still have to have code doing the kind of "manual callback" style I mentioned"

That's exactly right, specifically for steps. We considered other ways to wrap the workflow calls (so you don't have to do dbos.RunWorkflow(yourFunction)), but they got in the way of providing compile time type checking.

As Qian said, under the hood the Golang SDK is an embedded orchestration package that just requires Postgres to automate state management.

For example, check the RunWorkflow implementation: https://github.com/dbos-inc/dbos-transact-golang/blob/0afae2...

It does all the durability logic in-line with your code and doesn't rely on an external service.

Thanks for taking the time to share your insights! This was one of the most interesting HN comment I've seen in a while :)

It is very good at doing the job that people over-eagerly offload to specialized services. Queues, notifications, scheduled jobs. And can be specialized with extensions.