Check this out: https://github.com/graphile/worker/blob/3540df5ab4eb73f846d5...

Just grabbing the next available task to work on.

It makes the job-fetching more atomic, as the article points out properly. If you take >1, you have to sort the situation where 1 or more jobs fail, which adds an unreasonable amount of complexity.

On my first go I was taking batches, on the reasoning that it optimized for query performance. It was a complete disaster and I quickly realized that I was engaging in premature optimization.

Batching is out, stream-processing is in. If you design your job table correctly, PG will perform very well as an advanced stream processor.

I should really get around to writing some docs for this, but you may find this useful:

https://github.com/davidbanham/kewpie_go

I’ve been using it for years across multiple products.

I also built this for a client lately. It lets them interop their Haskell services with kewpie a bit easier without needing to write a native wrapper.

https://github.com/paidright/kewpie-http

Right, I was working mostly off one very well-written article on the topic, it was an easy search because there is really only one correct way to implement atomic job-handling with PG.