Designing a Passively Safe API
danealbaugh.com
danealbaugh.com
Now, any system I’ve seen designed around exactly once is complete garbage.
Not entirely true, DBOS can guarantee exactly once execution if your API calls are idempotent.
I thought that was what 'idempotent' meant.
You don't have idempotent crashes.
The concept originates in maths, where it's functions that can be idempotent. The canonical example is projection operators: if you project a vector onto a subspace and then apply that same projection operator again you get the same vector again. In computing the term is sometimes used fairly loosely/analogistically like in the light switch example above. Sometimes, though, there is a mathematical function involved that is idempotent in the mathematical sense.
A form of idempotence is implied in "retries ... can't produce duplicate work" in the quote, but it isn't the whole story. Atomicity, for example, is also implied by the whole quote: the idea that an operation always either completes in its entirety or doesn't happen at all. That's independent of idempotence.
Transactional outboxes specifically are one of my favorite patterns: they’re not too hard to add and don’t require changing many core invariants of your system. If you already use some sort of message bus or queue, making publishes to it transactional under a given RDBMS is often as simple as adding some client side code and making sure that logical message deduplication and is present where appropriate: https://microservices.io/patterns/data/transactional-outbox....
If you use a separate message broker (Kafka, SQS, RabbitMQ) with this pattern, you’ll also need a sweeper cron job to re-dispatch failed publishes from the outbox table(s) as well.
Bonus points if this can be implemented on top of existing trigger-based audit table functionality.
send a() send b()
And know both will be sent at least once, without having to introduce an outbox and re-architect your code to use a message relay. We can nitpick the details, but being able to "just write normal code" and get strong guarantees is, imo, real progress.
I get that it is particularly valuable in that scenario by treating other services as "external API", but monolith also do call "external API" and delegate work to async tasks. The principles discussed here API are interesting beyond just micro-services while being lighter and simpler than Durable Execution.
Didn't we get to the point where we realized that microservices cause too much trouble down the road?
That's a largely ignorant opinion to have. Like any architecture, microservices have clear advantages and tradeoffs. It makes no sense to throw vague blanket statements at an architure style because you assume it "causes trouble", particularly when you know nothing about requirements or constraints and all architectures are far from bullet proof.
I am particularly not a fan of doing unnecessary work/over engineering, e.g. see https://charemza.name/blog/posts/agile/over-engineering/not-..., but even I think that sometimes things _are_ worth it
Being prepared for these things to happen and having code in place to automatically prevent, recognize and resolve these errors will keep you, the customers and everyone in between sane and happy.
You also need to think about what it means to double-charge your customers, what it means to them and their wallets, and to their relationship to you. Do you want their repeat business? What sums are we talking about? How do you find out about these double-charges, and how quickly? Do the customers have to complain to you first, or did you anticipate the problem and have things in place to flag these charges?
Yes, you can hire people in place of the code you didn't write, but that only makes sense if continuing to pay them is cheaper than writing the code once and then maintaining it, which also probably means the manual work generated should not scale in proportion with your business.
Finally, developing for more than the happy-path is not overengineering, it's plain old engineering. There is a point, a kind and size of business, where it makes sense to do these things properly, and then TFA comes into play. The cost of just winging it goes up and up, until you need to do something about it.
Strong disagree. Addressing expectable failure modes is not over engineering. It's engineering. How do you put together a system without actually thinking through how it fails, and how to prevent those failure scenarios from taking down your whole operation?
I have bad news for everyone. Nothing in computing is synchronous. Every instance we pretend it’s not and call it something else you have a potential failure under the right circumstances.
The more your design admits this the safer it will be. There are practical limits to this which you have to determine for yourself.
I think you need to sit this one out. This sort of vacuous pedantry does no one any good, and ignores that it's perfectly fine to model and treat some calls are synchronous, including plain old HTTP ones. Just because everything is a state machine this does not mean you buy yourself anything of value by modeling everything as a state machine.
TCP connect(3) is synchronous. Making a directory on a local filesystem is synchronous. fsync(2) is synchronous. Committing an RDBMS transaction is synchronous.
Saying “technically commands are posted and later observed, and that’s how we get security vulns” is an extraordinary claim. The vast, vast majority of program statements do not work thus. Memory operations immediately move instructions through the CPU to access data. Many IO operations request that the kernel immediately send an interrupt to a hardware device and wait for a response from that device to be written to memory—that’s synchronicity in the domain of electrical engineering, not just software. And sure, there’s periodicity and batching there (waiting for scheduler ticks and interrupt poll frequency and such), but none of that makes something less than synchronous; it just might slow it down. Unless you were referring only to timing attacks in your claim that security vulnerabilities result from not-really-synchronous actions, then I think that’s wrong twice over.
To expand on the examples: mkdir(2)’s durability (which is what we’re talking about when we refer to “synchronous-ness” of filesystem ops) depends—it depends on the filesystem and caching configuration of the system. But on many (I’d hazard most) file systems and configurations, new directories are persisted either immediately through the dentcache to the disk, or are during one of the next two calls to fsync(2), the next example I listed. And sure, there’s subtlety there! Your disk can lie, your RAID controller can lie and leave data in a write cache, exactly what is synchronous when you call fsync(2) depends on whether it’s the first or second call made in succession, and so on. But the fact remains that those calls do, in fact, block on the requested changes being made in many/most configurations. That’s far from your initial claim that “nothing is synchronous”.
Then consider the network examples. A connect(2) call isn’t like a socket send or filesystem write that might be cached or queued; that connect call blocks until a TCP negotiation with the target host’s network stack is performed. There’s an asterisk there as well (connection queues can make the call take longer, and we can have a semantic debate about atomicity vs synchronicity or “waiting for the thing to start” vs “waiting for the thing to finish” if you like), but the typical behavior of this action meets the bar for synchronicity as most people understand it.
The same is true further up the stack. A COMMIT RPC on most RDBMSes will indeed synchronously wait for the transaction’s changes to be persisted, or rolled back, to the database. The asterisks there are in the domain of two generals/client-server atomicity, or databases whose persistence settings don’t actually wait for things to be written to disk, but again: the majority of cases do, in fact, operate synchronously.
If “nothing is synchronous”, then how does read causality work? Like, my code can be relying on a hundred tiers of ephemeral and deceptive caches and speculatively executing out the ass, but a system call to read data from an IO source must necessarily be synchronous if I can observe that read when it finishes (either by seeing a read pointer advance on a file handle in the kernel, or just by using the data I read).
So … yeah, no. There’s nuance there, certainly. Deceptive and easy to mess up behavior, sure. But if that’s the complaint, say it—say “people do not understand the page cache and keep causing data loss because they assume write(2) works a certain way”. Say “people make wrong assumptions about the atomicity of synchronous operations”. Don’t say “nothing is synchronous”, because it isn’t true.
See https://rcrowley.org/2010/01/06/things-unix-can-do-atomicall..., https://datatracker.ietf.org/doc/html/rfc793.html#section-3...., and that amazing old … I think it was a JWZ article that compared data loss characteristics of Linux file systems given different power loss/fsync scenarios. Google isn’t helping me to find it, but perhaps someone who has it handy could link it here (unformatted HTML post that largely boiled down to “lots of drives and filesystems have durability issues; XFS and fsync are a good combination to achieve maximum durability”).