abelanger

Whoops sorry, I'll need to size this down but: https://gist.github.com/abelanger5/c513ef259b1b03fe2e5a27b96...

Going to create a PR into https://github.com/knobiknows/all-the-bufo later today.

If you really need to merge and deploy, I've been having some luck getting things moving with the gh cli:

    gh pr review <xx> --approve
    gh pr merge <xx> --squash --delete-branch

Might not help your deploy process if webhooks are degrading, but at least it can keep your remote in sync with local changes, if you need to deploy from local.

My wife and I have been playing this game every day for the past few weeks or so. Some of the daily levels are incredibly satisfying.

We have very different ways of approaching the harder problems; some of the clues are easier for someone with a formal CS/engineering background, but others just require a better memory of the different constraints, and she's much better at building a tagging system around those constraints.

Previously launched here: https://news.ycombinator.com/item?id=43923843

Hatchet | Product Engineer | NYC or SF or REMOTE (US and EU) | https://hatchet.run

Hey HN! I'm Alexander, one of the founders of Hatchet. Hatchet is an open-source platform for running background jobs at scale.

We're hiring product engineers who are excited to build the next class of engineering primitives, starting with queues, background tasks and durable execution. We started in early 2024 after launching our distributed task queue (https://news.ycombinator.com/item?id=39643136).

Hatchet is currently used by thousands of engineers for all kinds of workloads: log ingestion pipelines, code review agents, video encoding, GPU scheduling, etc. Our target customer is fast-growing startups who have a strong need for background jobs system. These days, that tends to be AI companies, though we're general-purpose and not exclusively targeted for AI workloads.

Stack: Postgres, Go, Typescript, React, Kubernetes

Applying: I'm trying something different this time. `ssh hatchet-jobs.com`

Note that we're fully open-source, which you can check out here: https://github.com/hatchet-dev/hatchet (and if you have thoughts / opinions / questions about the codebase, please include those in your note!)

> I do wish the guide was expanded a bit with lots of specific examples and details... a lot of it is hand-wavy blurbs.

I appreciate the feedback; I'm usually someone who tends to go into way too much detail, so this was difficult to write - I tried to focus on the "mental model" of understanding Postgres rather than very nuanced specifics. I tried to link out to my favorite articles on a number of subjects, and the Postgres manual is quite good.

Some external links from the article:

- https://www.digitalocean.com/community/tutorials/database-no...

- https://www.cybertec-postgresql.com/en/benefits-of-a-descend...

- https://martinfowler.com/bliki/ParallelChange.html

- https://www.cybertec-postgresql.com/en/tuning-autovacuum-pos...

Some internal links on where I've gone into our own use-cases in more detail:

- https://hatchet.run/blog/multi-tenant-queues (PG-backed queues)

- https://hatchet.run/blog/postgres-partitioning (PG partitioning)

(edit: formatting)

OP here, I appreciate this.

> in addition to minimizing locked records, make sure your locks are ordered deterministically across all queries (eg by id asc, always) or you’ll deadlock (but postgres has a really good deadlock detector so you’ll more likely just error out if you’re lucky)

This is really good advice, I should put this somewhere in the guide. To add to this, not only can you deadlock by not having a consistent `ORDER BY` when you're locking sets of rows, but you should also be careful of locking rows on tables in different orders. For example, even if you lock each row in a table with an ORDER BY and FOR UPDATE, if one tx locks `table_a` and then `table_b`, and the other locks `table_b` and then `table_a`, you'll deadlock. This is obvious in theory but exponentially harder to debug in practice, because you need to be globally aware of every table that a write touches - something that's bitten us in particular with certain extensions.

> learn about GIN (and GIST) indexes

We're just testing GIN for fast key-value lookups for JSONB columns, and the performance improvements have been really massive. Interestingly there was a large performance skew between AND vs OR on these key-value queries.

OP here, I appreciate the feedback! I tried to focus on things which could take down your database, so things like particularly slow reads and writes, autovacuum settings, reducing lock contention, particularly focused on cases that I've seen. There are lots of things that I left out which would belong in a general user guide.

We're heavy users of stored functions because we're (perhaps overly) reliant on Postgres triggers, which can improve performance by reducing network round-trips but are fairly risky because they're difficult to monitor and observe.

Hatchet | Full-Stack Engineer | NYC or SF or REMOTE (US and EU) | https://hatchet.run

Hey HN! I'm Alexander, one of the founders of Hatchet. Hatchet is an open-source platform for running background jobs at scale.

We're hiring engineers who are excited to build the next class of engineering primitives, starting with queues, background tasks and durable execution. We started in early 2024 after launching our distributed task queue (https://news.ycombinator.com/item?id=39643136).

Hatchet is currently used by thousands of engineers for all kinds of workloads: log ingestion pipelines, code review agents, video encoding, GPU scheduling, etc. Our target customer is fast-growing startups who have a strong need for background jobs system. These days, that tends to be AI companies, though we're general-purpose and not exclusively targeted for AI workloads.

Stack: Postgres, Go, Typescript, React, Kubernetes

Applying: email me at alexander@hatchet.run and tell me about something impressive you've built, along with your CV and why you're interested in Hatchet.

Note that we're fully open-source, which you can check out here: https://github.com/hatchet-dev/hatchet (and if you have thoughts / opinions / questions about the codebase, please include those in your note!)

> Our analysis revealed the fork-to-star ratio as the strongest simple heuristic for identifying potential manipulation. The logic is straightforward: a star costs nothing and conveys no commitment. A fork means someone downloaded the code to use or modify it.

This is just clearly...incorrect? You can both modify code without forking it and most software is distributed via a registry or binary download, which also wouldn't be represented in forks. For most projects, the number of forks is a lossy signal for how busy the contributor ecosystem is, nothing else.

Candidly we're still trying to figure that out: all of the plumbing is there in the open source, but the actual implementation of writes to S3 are only on the cloud version. This is partially because we're loath to introduce additional dependencies, and partially because this job requires a decent amount of CPU and memory and would have to run separate from the Hatchet engine, which adds complexity to self-hosted setups. That said, we're aware of multi-TB self-hosted instances, and this would be really useful for them - so it's important that we can get this into the open source.

The payloads are time-partitioned (in either case) so we do drop them after the user-defined retention period.

Author here - I'm also generally skeptical of coding agents, but with the right problem domain and approach they can produce quality output when paired with humans. There was a point in time in the chess world where computer + human was stronger than computer or human alone. I think we're in that era for a handful of applications. Not for things like kernels, browsers, or databases.

> Besides, who is going to maintain that code?

I maintain the code. If Claude gets sunset tomorrow, I'll still be able to maintain and write it - I've already rewritten parts of it.

You could make the same argument for a team member leading a project that you've worked on. Is that code forever required to be maintained by one team member?

Previously the overhead of ensuring code quality when the development process was driven by Claude Code was greater than just writing the code myself. But that was different for this project.

Hi, thanks! To be clear, the demo there is merely a WASM-based Ghostty build which is rendering the TUI on a web page, just so people could try it out without needing to install anything. The actual TUI runs in your terminal. I'm guessing it's the WASM side of things causing the fans to spin, which you wouldn't see locally.

Hi everyone, I enjoyed building this TUI for myself and wanted to write down how I did it. I appreciate all the thoughts and feedback! The web app is our main investment, but I think there's a slice of developers who really like to interact with TUIs, so I'm going to keep working on it.

For the demo at https://tui.hatchet.run, to answer some messages asking about it: I built this with the fantastic ghostty-web project (https://github.com/coder/ghostty-web). It's been a while since I've used WASM for anything and this made it really easy. I deployed the demo across six Fly.io regions (hooray stateless apps) to try to minimize the impact of keystroke latency, but I imagine it's still felt by quite a few people.

You're right - I'll remove that now until we can get it more performant or drop it altogether. This wasn't something we caught during testing. I appreciate the feedback!

Sounds like the CodeRabbit controversy: https://xcancel.com/harjotsgill/status/2004050004785484172

Drawing the boundary at high throughput, huge fan-out and ultra-low-latency is correct - I'd also add that MQs are often used for pub/sub and signaling.

MQs are heavily optimized for reducing E2E latency between publishers and consumers in a way that DE engines are not, since DE engines usually rely on an ACID compliant database. Under load I've seen an order of magnitude difference in enqueue times (low single-digit milliseconds for the MQ p95 vs 10ms p95 for Postgres commit times). And AMQP has a number of routing features built-in (i.e. different exchange types) that you won't see in DE engines.

Another way to think about it is that message queues usually provide an optional message durability layer alongside signaling and pub/sub. So if you need a very simple queue with retries _and_ you need pub/sub, I'd be eyeing an MQ (or a DE execution engine that supports basic pub/sub, like Hatchet).

I wrote about our perspective on this here: https://hatchet.run/blog/durable-execution

( disclaimer - I'm one of the people behind https://github.com/hatchet-dev/hatchet )

Not harmful for now - "fork from checkpoint" would be perfectly fine for me at the moment. The main issue (as flagged in the post) is that setting up additional tooling can take a while!

In the longer term, docker is nice from a reproducibility + CI perspective, and a docker build is already something can easily work with and track in my system.

One thing I've heard but not verified with other sandboxed execution providers is that startup times for custom images can be quite slow, so it could be a potential differentiator given Fly's existing infra.

This is seriously cool - it's exactly the DX and API I've been waiting for from sandboxed execution providers.

I'd love to be able to configure the base image/VM in a way that doesn't bundle coding tools or anything else I don't need, and comes with some other binaries installed (I'm more interested in using this as an API for a sandbox use-case I have). Is there a way to do this at the moment / is this on the roadmap?

Another option would be configuring the sprite via checkpoint and then cloning the checkpoint from a base sprite, but I don't see this option anywhere either.

Yep! Feel free to email me at alexander [at] hatchet [dot] run

Oh no, haven't gotten reports about the website being slow, thanks for flagging! Which browser are you using?

Regarding Temporal, our goal is to create the best developer experience possible, and we started Hatchet because we felt that Temporal misses the mark (I used Temporal for years before starting Hatchet).

The primary difference is that we're not solely focused on durable workflows, we're a general-purpose background jobs platform which offers durable workflows as a feature. In our view there are a set of equally important primitives: tasks, events, streaming/pubsub, concurrency, priority, rate limiting, scheduling, and yes, durable workflows.

Tasks being the entrypoint to the platform, rather than immediately dealing with the overhead of durable workflows, generally makes Hatchet easier to adopt for engineering teams. I wrote a little more about how task queues relate to durable execution here: https://hatchet.run/blog/durable-execution

We've also invested quite heavily in platform features like logging, observability, alerting, and our UI which either aren't offered or are underdeveloped in Temporal.

But ultimately I'd encourage people to give both a try - we're both MIT licensed and can easily be run locally.

Hatchet | Founding Engineer | NYC or REMOTE (US and EU) | https://hatchet.run

Hey HN! I'm Alexander, one of the founders of Hatchet. Hatchet is an open-source platform for running background jobs at scale.

We're hiring engineers who are excited to build the next class of engineering primitives, starting with queues, background tasks and durable execution. We started in early 2024 after launching our distributed task queue (https://news.ycombinator.com/item?id=39643136).

Hatchet is currently used by thousands of engineers for all kinds of workloads: log ingestion pipelines, code review agents, video encoding, GPU scheduling, etc. Our target customer is fast-growing startups who have a strong need for background jobs system. These days, that tends to be AI companies, though we're general-purpose and not exclusively targeted for AI workloads.

Stack: Postgres, Go, Typescript, React, Kubernetes

Apply here: https://www.ycombinator.com/companies/hatchet-run/jobs/SNpCm...

Or email me at alexander [at] hatchet [dot] run

I mentioned this towards the bottom of the post, but to reiterate: we're extremely grateful to Laurenz for helping us out here, and his post on this is more than worth checking out: https://www.cybertec-postgresql.com/en/partitioned-table-sta...

(plus an interesting discussion in the comments of that post on how the query planner chose a certain row estimate in the specific case that Laurenz shared!)

The other thing I'll add is that we still haven't figured out:

1. An optimal ANALYZE schedule here on parent partitions; we're opting to over-analyze than under-analyze at the moment, because it seems like our query distribution might change quite often.

2. Whether double-partitioned tables (we have some tables partitioned by time series first, and an enum value second) need analyze on the intermediate tables, or whether the top-level parent and bottom-level child tables are enough. So far just the top-level and leaf tables seem good enough.

FWIW, I think the memory snapshotting idea isn't going to work for most stacks for a few different reasons, but to speak more broadly on API design for durable execution systems, I agree completely. One of the issues with Temporal and Hatchet in its current state is that it currently abstracts concepts that are essential for the developer to understand, like what it means for a workflow to be durable, while the developer is building the system. So you end up discovering a bunch of weird behaviors like "non-determinism error" when starting to test these systems without a good grasp of the fundamentals.

We're investing heavily in separating out some of these primitives that are separately useful and come together in a DE system: tasks, idempotency keys and workflow state (i.e. event history). I'm not sure exactly what this API will look like in its end state, but idempotency keys, durable tasks and event-based histories are independently useful. This is only true of the durable execution side of the Hatchet platform, though; I think our other primitives (task queues, streaming, concurrency, rate limiting, retries) are more widely used than our `durableTasks` feature because of this very problem you're describing.

Sure, I'll bite. Task-level idempotency is not the problem that durable execution platforms are solving. The core problem is the complexity that arises when one part of your async job becomes distributed: the two common ones are distributed runtime (compute) and distributed application state.

Let's just take the application state side. If your entire async job can be modeled as a single database transaction, you don't need a durable execution platform, you need a task queue with retries - our argument at Hatchet is that this is many (perhaps most) async workloads, which is why the durable task queue is the primary entrypoint to Hatchet, and durable execution is only a feature for more complex workloads.

But once you start to distribute your application state - for example, different teams building microservices which don't share the same database - you have a new set of problems. The most difficult edge case here is not the happy path with multiple successful writes, it's distributed rollbacks: a downstream step fails and you need to undo the upstream step in a different system. In these systems, you usually introduce an "orchestrator" task which catches failures and figures out how to undo the system in the right way.

It turns out these orchestrator functions are hard to build, because the number of failure scenarios are many. So this is why durable execution platforms place some constraints on the orchestrator function, like determinism, to reduce the number of failure scenarios to an amount easy to reason about.

There are other scenarios other than distributed rollbacks that lead to durable execution, it turns out to be a useful and flexible model for program state. But this is a common one.

The "constraining functions to only be durable" idea is really interesting to me and would solve the main gotcha of the article.

It'd be an interesting experiment to take memory snapshots after each step in a workflow, which an API like Firecracker might support, but likely adds even more overhead than current engines in terms of expense and storage. I think some durable execution engines have experimented with this type of system before, but I can't find a source now - perhaps someone has a link to one of these.

There's also been some work, for example in the Temporal Python SDK, to overwrite the asyncio event loop to make regular calls like `sleep` work as durable calls instead, to reduce the risk to developers. I'm not sure how well this generalizes.

Hah, well I'll avoid _talking to_ vendors, more specifically I'll avoid talking to salespeople selling a technical product until we're pretty deep in the product. I do tend to not use vendors that don't have a good self-serve path or mechanism to get my technical questions answered.

If anyone needs commands for turning off the CF proxy for their domains and happens to have a Cloudflare API token.

First you can grab the zone ID via:

    curl -X GET "https://api.cloudflare.com/client/v4/zones" -H "Authorization: Bearer $API_TOKEN" -H "Content-Type: application/json" | jq -r '.result[] | "\(.id) \(.name)"'

And a list of DNS records using:

    curl -X GET "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" -H "Authorization: Bearer $API_TOKEN" -H "Content-Type: application/json"

Each DNS record will have an ID associated. Finally patch the relevant records:

    curl -X PATCH "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID" -H "Authorization: Bearer $API_TOKEN" -H "Content-Type: application/json" --data '{"proxied":false}'

Copying from a sibling comment - some warnings:

- SSL/TLS: You will likely lose your Cloudflare-provided SSL certificate. Your site will only work if your origin server has its own valid certificate.

- Security & Performance: You will lose the performance benefits (caching, minification, global edge network) and security protections (DDoS mitigation, WAF) that Cloudflare provides.

- This will also reveal your backend internal IP addresses. Anyone can find permanent logs of public IP addresses used by even obscure domain names, so potential adversaries don't necessarily have to be paying attention at the exact right time to find it.

Hatchet (https://hatchet.run) | New York City (IN-PERSON | REMOTE) | Full-time

We're hiring for a number of engineering positions to help us with development on our open-source, distributed task queue: https://github.com/hatchet-dev/hatchet.

We launched on HN several times last year, you can check out our launches here https://news.ycombinator.com/item?id=39643136 and here https://news.ycombinator.com/item?id=40810986. We're two second-time YC founders in this for the long haul. Since our launches we now process over 50 million tasks/day on our cloud platform.

As an early engineer at Hatchet, you'll be responsible for contributing across the entire codebase. We'll compensate accordingly and with high equity. All team members are technical and contribute code.

Stack: Typescript/React, Go and PostgreSQL.

To apply, email alexander [at] hatchet [dot] run, and include the following:

1. Tell us about something impressive you've built.

2. Ask a question or write a comment about the state of the project. For example: a file that stood out to you in the codebase, a Github issue or discussion that piqued your interest, a general comment on distributed systems/task queues, or why our code is bad and how you could improve it.

Or apply online here: https://www.ycombinator.com/companies/hatchet-run/jobs

If you don't think you are a good fit for one of the roles, but you're interested in developer tools or infrastructure, please reach out anyway - we'll still consider your application!

One thing I've been wondering recently: has the experience of using software (specifically web apps) been getting better? It seems like a natural extension of significantly increased productivity would lead to fewer buggy websites and apps, more intuitive UIs, etc.

Linear was a very early-stage product I tested a few months after their launch where I was genuinely blown away by the polish and experience relative to their team size. That was in 2020, pre-LLMs.

I have yet to see an equally polished and impressive early-stage product in the past few years, despite claims of 10x productivity.