abelanger

You can get around some of this coordination by doing more work locally, in-process -- but you're risking the availability of your primary API. The work that you're doing as a background job or workflow may:

1. Be unbounded -- 1 API call may correspond to thousands or hundreds of thousands of reads or writes

2. Be resource intensive, on either the database side by eating up connections which are needed by your API, or by blocking your event loop/process, particularly in Python and Typescript

> Once you get to the scale that breaks a library based system like DBOS, you need to move away from a central coordinator

There are different levels of coordination here. At some point, workers are always going to have to coordinate "who is working on what" -- whether that's by placing locks and using status updates in the database, using a traditional queue, or using a central server like you described with DBOS Cloud. The same goes for a dedicated orchestrator -- those can also partition by "who is working on which workers." In other words, a dedicated service can also be decentralized, these don't seem mutually exclusive.

To make this more concrete -- let's say, for example, you have hundreds of workers on a library based system like DBOS which all correspond to individual connections and transactions to read/write data. Whereas in a system like Temporal or Hatchet, you might have 10s of nodes to support 1000s of workers, with the ability to bulk enqueue and dequeue more work, which will increase throughput and reduce DB load rather significantly. You'd lose (most of) the benefits of bulk writes in a library based system.

Definitely agree that the dev experience is better with a library, particularly for lightweight and low-volume tasks (Hatchet is also moving in the same direction, we'll be releasing library-only mode this month). And I really like the transactional safety built into DBOS!

My concern is as you start to see higher volume, more workers, or load patterns that don't correspond to your primary API. At that point, a dedicated database and service to orchestrate tasks starts to become more necessary.

Disclaimer: I'm a co-founder of Hatchet (https://github.com/hatchet-dev/hatchet), which is a Postgres-backed task queue that supports durable execution.

> Because a step transition is just a Postgres write (~1ms) versus an async dispatch from an external orchestrator (~100ms), it means DBOS is 25x faster than AWS Step Functions

Durable execution engines deployed as an external orchestrator will always been slower than direct DB writes, but the 1ms delay versus ~100ms doesn't seem inherent to the orchestrator being external. In the case of Hatchet, pushing work takes ~15ms and invoking the work takes ~1ms if deployed in the same VPC, and 90% of that execution time is on the database. In the best-case, the external orchestrator should take 2x as long to write a step transition (round-trip network call to the orchestrator + database write), so an ideal external orchestrator would be ~2ms of latency here.

There are also some tradeoffs to a library-only mode that aren't discussed. How would work that requires global coordination between workers behave in this model? Let's say, for example, a global rate limit -- you'd ideally want to avoid contention on rate limit rows, assuming they're stored in Postgres, but each worker attempting to acquire a rate limit simultaneously would slow down start time significantly (and place additional load on the DB). Whereas with a single external orchestrator (or leader election), you can significantly increase throughput by acquiring rate limits as part of a push-based assignment process.

The same problem of coordination arises if many workers are competing for the same work -- for example if a machine crashes while doing work, as described in the article. I'm assuming there's some kind of polling happening which uses FOR UPDATE SKIP LOCKED, which concerns me as you start to scale up the number of workers.

I've found that I rely most heavily on LLMs when:

1. I'm developing a utility package that's easily testable and I'm certain of the interface. I'll write the interface for the package in my editor, then ask an LLM to generate unit tests. Then, I'll sketch out the function calls/structure of the package and get an LLM to fill out the rest.

2. I'm bug bashing and want to quickly check if the bug is obvious. I'll feed a description of the behavior into GPT/Claude along with the relevant code (generally as little code as possible to prevent listicle-type responses that are irrelevant).

3. I'm adding code that follows an established pattern within the codebase -- for example, adding a new API handler that involves generating an OpenAPI path + component snippet, an http handler, and a few database methods. This is when copilots are particularly useful.

4. I'd like a sanity check for issues in a complex bit of code I've written.

I find these mirror the tasks you'd typically hand off to a less experienced dev on the team -- these are things that require validation based on knowledge that you already have, and the validation is more efficient than doing it yourself.

It's not clear how many records were generated in the sample dataset for these benchmarks. The methodology [1] and Github repo [2] show how to seed tables of 1k records, but it's not clear what size the results on the website were using.

Additionally, it seems that these benchmarks were run by executing queries serially, which isn't particularly interesting to me -- I'm more interested in what latency and resource consumption look like for each query under any type of load. The issues that we were seeing with Prisma [3] were latency in acquiring a connection and executing a query when many queries were running simultaneously (with idle connections available in the pool and a high connection limit on Postgres). I'd also be curious about the difference in performance for something like a nested `updateMany` where depending on how the query is generated, could deadlock or place a lock on more rows than necessary -- and where the generated queries actually matter.

Running queries serially against what is presumably 1k records per table doesn't seem particularly valuable.

[1] https://www.prisma.io/blog/performance-benchmarks-comparing-...

[2] https://github.com/prisma/orm-benchmarks/

[3] https://docs.hatchet.run/blog/migrating-off-prisma

Hatchet (https://hatchet.run) | New York City (IN-PERSON) | Full-time

We're hiring a founding engineer to help us with development on our open-source, distributed task queue: https://github.com/hatchet-dev/hatchet.

We recently launched on HN, you can check out our launches here https://news.ycombinator.com/item?id=39643136 and here https://news.ycombinator.com/item?id=40810986. We're two second-time YC founders in this for the long haul and just wrapped up the YC W24 batch.

As a founding engineer, you'll be responsible for contributing across the entire codebase. We'll compensate accordingly and with high equity. It's currently just the two founders + a part-time contractor. We're all technical and contribute code.

Stack: Typescript/React, Go and PostgreSQL.

To apply, email alexander [at] hatchet [dot] run, and include the following:

1. Tell us about something impressive you've built.

2. Ask a question or write a comment about the state of the project. For example: a file that stood out to you in the codebase, a Github issue or discussion that piqued your interest, a general comment on distributed systems/task queues, or why our code is bad and how you could improve it.

Yep, we have support for "retry 5 times, then give up" (https://docs.hatchet.run/home/features/retries/simple) and "text me" - you can use either our built-in alerting features which integrate with email and Slack, or configure your own on failure step (https://docs.hatchet.run/home/features/on-failure-step).

Configurable retry delays are currently in development.

You're right, if all you need is a queue with a small number of workers connected at low volume, you don't need Hatchet or any other managed queue - you can get some pretty performant behavior with something like: https://github.com/abelanger5/postgres-fair-queue/blob/main/....

The point of Hatchet is to support more complex behavior - like chaining tasks together, building automation around querying and retrying failed tasks, handling a lot of the fairness and concurrency use-cases you'd otherwise need to build yourself, etc - or just getting something that works out of the box and can support those use-cases in the future.

And if you are running at low volume and trying to debug user issues, a grafana panel isn't going to get you the level of granularity or admin control you need to track down the errors in your methods (rather than just at the queue level). You'd need to integrate your task queue with Sentry and a logging system - and in our case, error tracing and logging are available in the Hatchet UI.

Thanks! Yes, our recommended approach is to write a parent workflow which calls child workflows registered on a different worker. We have users who are managing a set of Python functions from a Typescript backend with this approach.

It's also possible to have a single DAG workflow (instead of parent/child) that has steps across multiple languages, but you'll need to use a relatively undocumented method called `RegisterAction` within each SDK and use the API to register the DAG (instead of using the built-in helpers) for this use-case. So we recommend using the parent/child workflows instead.

Very cool! Does it support the latest version of Celery?

And to answer the question, no, the license doesn't restrict a company from offering a hosted version of Hatchet. We chose the license that we'd want to see if we were making a decision to adopt Hatchet.

That said, managing and running the cloud version is significantly from a version meant for one org -- the infra surrounding the cloud version manages hundreds and eventually thousands of different tenants. While it's all the same open-source engine + API, there's a lot of work required to distribute the different engine components in a way that's reliable and supports partitioning databases between tenants.

Yes, I remember reading the post and the discussion surrounding it being very high quality!

I particularly like the section on escape hatches - though you start to see the issue with this approach when you use something like Celery, where the docs and Github issues contain a number of warnings about using Redis. RabbitMQ also tends to be very feature-rich from an MQ perspective compared to Redis, so it gets more and more difficult to support both over time.

We'd like to build in escape hatches as well - this starts with the application code being the exact same whether you're on cloud or self-hosted - and adding support for things like archiving task result storage to the object store of your choice, or swapping out the pub/sub system.

We'd like to stabilize our existing 3 SDKs and create a proper spec for future SDKs to implement. While we use proto definitions and openapi to generate clients, there are a lot of decisions made while calling these APIs that are undocumented but kept consistent between TS, Python and Go.

Once that's done and we consider our core API stable, there's a good chance we'll start tackling a new set of SDKs later this year.

It does seem like some really great options are emerging in the Go community, and a lot of newer execution frameworks are supporting Go as one of the first languages. Another great addition is https://github.com/riverqueue/river.

While the execution model is very similar to Airflow, we're primarily targeting async jobs which are spawned from an application, while Airflow is primarily for data pipelines. The connector ecosystem of Airflow is very powerful and not something that we're trying to replace.

That's not to say you can't use Hatchet for data pipelines - this is a common use-case. But you probably don't want to use Hatchet for big data pipelines where payload sizes are very large and you're working with payloads that aren't JSON serializable.

Airflow also tends to be quite slow when the task itself is short-lived. We don't have benchmarks, but you can have a look at Windmill's benchmarks on this: https://www.windmill.dev/docs/misc/benchmarks/competitors#re....

If we’re going to give credit where credit’s due, the history of durable execution traces back to the ideas of AWS step functions and Azure durable functions alongside the original Cadence and Conductor project. A lot of the features here are attempting to make patterns in those projects accessible to a wider range of developers.

Hatchet is also event driven [1], has built-in support for tracing and metrics, and has a TS [2], Python [3] and Golang SDK [4], has support for throttling and rate limiting [5], concurrency with custom multi-tenancy keys [6], works on serverless [7], and supports procedural workflows [8].

That said, there are certainly lots of things to work on. Batching and better tracing are on our roadmap. And while we don’t have a Java SDK, we do have a Github discussion for future SDKs that you can vote on here: https://github.com/hatchet-dev/hatchet/discussions/436.

[1] https://docs.hatchet.run/home/features/triggering-runs/event...

[2] https://docs.hatchet.run/sdks/typescript-sdk

[3] https://docs.hatchet.run/sdks/python-sdk

[4] https://docs.hatchet.run/sdks/go-sdk

[5] https://docs.hatchet.run/home/features/rate-limits

[6] https://docs.hatchet.run/home/features/concurrency/round-rob...

[7] https://docs.hatchet.run/home/features/webhooks

[8] https://docs.hatchet.run/home/features/child-workflows

Re Inngest - there are a few differences:

1. Hatchet is MIT licensed and designed to be self-hosted in production, with cloud as an alternative. While the Inngest dev server is open source, it doesn't support self-hosting: https://www.inngest.com/docs/self-hosting.

2. Inngest is built on an HTTP webhook model while Hatchet is built on a long-lived, client-initiated gRPC connection. While we support HTTP webhooks for serverless environments, a core part of the Hatchet platform is built to display the health of a long-lived worker and provide worker-level metrics that can be used for autoscaling. All async runtimes that we've worked on in the past have eventually migrated off of serverless for a number of reasons, like reducing latency or having more control over things like runtime environment and DB connections. AFIAK the concept of a worker or worker health doesn't exist in Inngest.

There are the finer details which we can hash out in the other thread, but both products rely on events, tasks and durable workflows as core concepts, and there's a lot of overlap.

That's correct, you can only create tasks via the gRPC client, Hatchet can't hook into the same transaction as your inserts or updates.

It seems like a very lightweight tasks table in your existing PG database which represents whether or not the task has been written to Hatchet would solve both of these cases. Once Hatchet is sent the workflow/task to execute, it's guaranteed to be enqueued/requeued. That way, you could get the other benefits of Hatchet with still getting transactional enqueueing. We could definitely add this for certain ORM frameworks/SDKs with enough interest.

While I understand the sentiment, we see it very differently. We're interested in creating the best product possible, and being open source helps with that. The users who are self-hosting in our Discord give extremely high quality feedback and post feature ideas and discussions which shape the direction of the product. There's plenty of room for Hatchet the OSS repo and Hatchet the cloud version to coexist.

> develop all the functionality of RabbitMQ as a Postgres extension with the most permissive license

That's fair - we're not going to develop all the functionality of RabbitMQ on Postgres (if we were, we probably would have started with a amqp-compatible broker). We're building the orchestration layer that sits on top of the underlying message queue and database to manage the lifecycle of a remotely-invoked function.

> My main thing is the RabbitMQ dependency (that seems to be a topic of interest in this thread). Getting rid of that and just depending on PG seems like the main path forward that would increase adoption.

Yep, we agree - this is more a matter of bandwidth as well as figuring out the final definition of the pub/sub interface. While we wouldn't prefer to maintain two message queue implementations, we likely won't drop the RabbitMQ implementation entirely, even if we offer Postgres as an alternative. So if we do need to support two implementations, we'd prefer to build out a core set of features that we're happy with first. That said, the message queue API is definitely stabilizing (https://github.com/hatchet-dev/hatchet/blob/31cf5be248ff9ed7...), so I hope we can pick this up in the coming months.

> You also compare yourself against Celery and BullMQ, but there is also talk in the readme around durable execution. That to me puts you in the realm of Temporal. How would you say you compare/compete with Temporal? Are you looking to compete with them?

Yes, our child workflows feature is an alternative to Temporal which lets you execute Temporal-like workflows. These are durable from the perspective of the parent step which executes them, as any events generated by the child workflows get replayed if the parent step re-executes. Non-parent steps are the equivalent of a Temporal activity, while parent steps are the equivalent of a Temporal workflow.

Our longer-term goal is to build a better developer experience than Temporal, centered around observability and worker management. On the observability side, we're investing heavily in our dashboard, eventing, alerting and logging features. On the worker management side, we'd love to integrate more natively with worker runtime environments to handle use-cases like autoscaling.

Yeah, pretty much - that was more of a side project while figuring out what to work on next. Plus the Terraform licensing changes were on the horizon and I became a little frustrated with the whole ecosystem.

Part of the reason for working on Hatchet (this version) was that I built the Terraform management tool on top of Temporal and felt there was room for improvement.

(for those curious - https://github.com/hatchet-dev/hatchet-v1-archived)

We're using RabbitMQ for pub/sub between different components of our engine. The actual task queue is entirely backed by Postgres, but things like streaming events between different workers are done through RabbitMQ at the moment, as well as sending a message from one component to another when you distribute the engine components. I've written a little more about this here: https://news.ycombinator.com/item?id=39643940.

We're eventually going to support a lightweight Postgres-backed messaging table, but the number of pub/sub messages sent through RabbitMQ is typically an order of magnitude higher than the number of tasks sent.

No, the whole task doesn't execute as a postgres transaction. Transactions will update the status of a task (and higher-order concepts like workflows) and assign/unassign work to workers, but they're short-lived by design.

For some more detail -- to ensure we can't assign duplicate work, we track which workers are assigned to jobs by using the concept of a WorkerSemaphore, where each worker slot is backed by a row in the WorkerSemaphore table. When assigning tasks, we scan the WorkerSemaphore table and use `FOR UPDATE SKIP LOCKED` to skip any locked rows help by other assignment transactions. We also have a uniqueness constraint on the task id across all WorkerSemaphores to ensure that no more than 1 task can be acquired by a semaphore.

This is slightly different to the way most pg-backed queues work, where `FOR UPDATE SKIP LOCKED` is done on the task level, but this is because not every worker maintains its own connection to the database in Hatchet, so we use this pattern to assign tasks across multiple workers and route the task via gRPC to the correct worker after the transaction completes.

Hatchet (https://hatchet.run) | New York City (IN-PERSON) | Full-time

We're hiring a founding engineer to help us with development on our open-source, distributed task queue: https://github.com/hatchet-dev/hatchet.

We recently launched on HN, you can check out our launch here: https://news.ycombinator.com/item?id=39643136. We're two second-time YC founders in this for the long haul and just wrapped up the YC W24 batch.

As a founding engineer, you'll be responsible for contributing across the entire codebase. We'll compensate accordingly and with high equity. It's currently just the two founders + a part-time contractor. We're all technical and contribute code.

Stack: Typescript/React, Go and PostgreSQL.

To apply, email alexander [at] hatchet [dot] run, and include the following:

1. Tell us about something impressive you've built.

2. Ask a question or write a comment about the state of the project. For example: a file that stood out to you in the codebase, a Github issue or discussion that piqued your interest, a general comment on distributed systems/task queues, or why our code is bad and how you could improve it.

Hatchet (https://hatchet.run) | New York City (IN-PERSON) | Full-time

We're hiring a founding engineer to help us with development on our open-source, distributed task queue: https://github.com/hatchet-dev/hatchet.

We recently launched on HN, you can check out our launch here: https://news.ycombinator.com/item?id=39643136. We're two second-time YC founders in this for the long haul and we are just wrapping up the YC W24 batch.

As a founding engineer, you'll be responsible for contributing across the entire codebase. We'll compensate accordingly and with high equity. It's currently just the two founders + a part-time contractor. We're all technical and contribute code.

Stack: Typescript/React, Go and PostgreSQL.

To apply, email alexander [at] hatchet [dot] run, and include the following:

1. Tell us about something impressive you've built.

2. Ask a question or write a comment about the state of the project. For example: a file that stood out to you in the codebase, a Github issue or discussion that piqued your interest, a general comment on distributed systems/task queues, or why our code is bad and how you could improve it.

Yeah we're using RabbitMQ for pub/sub (but are considering getting rid of it) and Postgres for the actual task queue. There's some more about that here: https://news.ycombinator.com/item?id=39643940.

Thank you! I've been using https://jitter.video with the Lottie exporter. It also has a Figma plugin so you can reuse components.

I created a gist with these recommendations - certainly an improvement, but I don't think it gets around the `WindowAgg` running across all 10k rows: https://gist.github.com/abelanger5/5c1a75755072239716cb587a2.... Does this accurately implement your suggestions?

Happy to try out other suggestions, but I haven't found a way to get a window function or `JOIN LATERAL` to scale in near-constant time for this queueing strategy.

Ah, great catch - I just pushed an update to match the query from the article. I wasn't looking at much except for the WindowAgg line, thank you!

I tried with a similar indexing strategy - it did make a very noticeable difference, breaking at about 40000 enqueued tasks instead of 25000. I left indexing out of the article because it can open up a different can of worms with performance degradation over a longer time horizon.

I also tried with an `ORDER BY RANDOM()` across group keys first, which does help with fairness but breaks the "deterministic" element.

No, this assumes you've already split your tasks into quanta of approximately the same duration - so all tasks are weighted equally in terms of execution time. If each of the tasks have different weights you might be looking at something like deficit round-robin [1], which I've looked at implementations for in RabbitMQ and we're thinking about how to implement in PG as well [2].

[1] https://www.cs.bu.edu/fac/matta/Teaching/cs655-papers/DRR.pd...

[2] https://nithril.github.io/amqp/2015/07/05/fair-consuming-wit...

Hatchet (https://hatchet.run) | New York City | Full-time

We're hiring a founding engineer to help us with development on our open-source, distributed task queue: https://github.com/hatchet-dev/hatchet.

We recently launched on HN, you can check out our launch here: https://news.ycombinator.com/item?id=39643136. We're two second-time YC founders in this for the long haul and we are just wrapping up the YC W24 batch.

As a founding engineer, you'll be responsible for contributing across the entire codebase. We'll compensate accordingly and with high equity. It's currently just the two founders + a part-time contractor. We're all technical and contribute code.

Stack: Typescript/React, Go and PostgreSQL.

To apply, email alexander [at] hatchet [dot] run, and include the following:

1. Tell us about something impressive you've built.

2. Ask a question or write a comment about the state of the project. For example: a file that stood out to you in the codebase, a Github issue or discussion that piqued your interest, a general comment on distributed systems/task queues, or why our code is bad and how you could improve it.