Building Reliable Distributed Systems in Node.js

Temporal is an implementation of a paradigm I got interested in back in 2019. I wasn’t at one of those companies that had heard about Cadence, so when I was searching around to see if anyone had actually already built this idea I’d come up with, I stumbled upon Zenaton. It’s no longer around, didn’t get PMF, so I was happy when Temporal came out of stealth mode a few months later - was nice to have my intuition in this area validated.

We’ve been using Temporal quite successfully in Go (and more recently Python) for a little while now. It could do with being a bit easier to get up and running with, but day-to-day usage is very nice. I don’t think I could go back to plain out message queues, this paradigm is a real time saver.

The biggest challenge is deciding how many things are nails for the hammer that is Temporal. You tend to start out using it to replace an existing mess of task orchestration; but then you realise its actually a pretty good fit for any write operation that can’t neatly work in a single database transaction (because it’s hitting multiple services, technologies, third parties etc).

You have to be careful to keep your workflows deterministic, but once you get used to the paradigm, it’s enjoyable.

This post talks about the durable execution systems, which include Azure Durable Functions, Amazon SWF, Uber Cadence, Infinitic, and Temporal.

Durable execution systems run our code in a way that persists each step the code takes. If the process or container running the code dies, the code automatically continues running in another process with all state intact, including call stack and local variables.

Durable execution makes it trivial or unnecessary to implement distributed systems patterns like event-driven architecture, task queues, sagas, circuit breakers, and transactional outboxes. It’s programming on a higher level of abstraction, where you don’t have to be concerned about transient failures like server crashes or network issues.

Never heard of durable execution until now, but I've wondered about it. When I write backend code, I have to keep asking myself "what happens if the server goes down during this line of code?" This is often an issue in the middle of a customer order, like the example here. I end up relying on the database for very many tiny little things, like recording the fact that the user initiated an order before I start to process it.

But how fast is this? IIRC each little insert in my DB was taking like 5ms, which would add up quickly if I were to spam it everywhere; I assume durable execution layers are better optimized for that. Do they really only snapshot before and after async JS calls, treating all other lines as hermetic and thus able to be rerun?

Or you could not use a scripting language and save 50 times the cost