Hacker News Reader

I agree. This design is incredibly strange, and seems to throw away basically all distributed systems knowledge. I'm glad folks are playing with different ideas, but this one seems off.

View on HN

No, this is a fairly common pattern called having an 'outbox' where the emission/enquing of your event/message/job is tied to the transaction completion of the relevant domain data.

We use this to ensure Kafka events are only emitted when a process succeeds, this is very similar.

So when the the business data transaction commit a notify event is raised and a job row is inserted. Out of bound job broker listens to a notify event of the job-table or polls the table skipping rows and takes work for processing?

Basically.

For our particular use case, I think we're actually not using notify events. We just insert rows into the outbox table and the poller re-emits as kafka events and deletes successfully emitted events from the table.

Either business data and job are committed or none of them. Then as you write, either polling or listening to an even worker, can pick it up. Bonus stuff, from implementation perspective, is that if worker selects row FOR UPDATE (locking the job from others to pick up) and dies, Postgres will release the lock after some time, making the job available for other workers.

Reply on news.ycombinator.com