The disaggregated write-ahead log (2023)

Note that I’m commenting in the spirit of “being ok to dream”, as the article promoted.

There’s only so much creativity available for software engineers to work around the fundamental constraints imposed by the lower levels. Software Engineering being a highly demanded skill, I strongly believe, is a reflection of how inefficient are the interlaces of standard architectures and processes. It reminds me of the Curse of Lisp, which, roughly, states that the reason for its lack of popularity and absence of large communities of developers stems from it being too powerful of a language, when compared, say, to Java. OSes are monstrosities struggling to keep pace with demand. The very concept and handling of the File is limiting and obsolete. It’s not an ideal building block for databases. Another is the fact that we have generalized compute but not storage. Memory and Storage are synonyms, so I imagine the ideal scenario is for both to be a single entity. I mean, the memory hierarchy should be flattened in the future. If this happens concurrently with an increase in capacity sufficient to emulate an infinite tape, then a number of today’s cutting edge software architectures will become relics — memories of a time when computers were not as cooperative and malleable as we needed them to be. The information revolution is just beginning, after all, and I absolutely love the fact that the path forward will be paved, primarily, by the collective effort of a multitude of creative minds, as always. Great article.

This sentence is doing a lot of work: "Hypothetical S2 does a bit more to simplify the layers above – it makes leadership above the log convenient with leases and fenced writes."

It'd be awesome to have a bit more transactional help from S3. You could go a long way with 'only update this object if the ETags on these other objects are still the same'. I know AWS doesn't want to turn S3 into a full database but some updates you just can't do without having a whole 2nd service running alongside to keep track of the states of your updates.

Consensus protocols, durability and transactional semantics are (should be) closely coupled. I recall TigerBeetle discussing somewhere how they could achieve better throughput and durability guarantees by combining replication/recovery with the consensus protocol, instead of layering it above. I.e. disaggregating the log can be expensive. There's a reference in [0] that might elaborate.

> TigerBeetle is “fault-aware” and recovers from local storage failures in the context of the global consensus protocol, providing more safety than replicated state machines such as ZooKeeper and LogCabin

[0] https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/DE...

Generic robust replicated log would be nice to have. For two products I've implemented leaderless oplog to back application level replication to sync not only database changes but also configuration files and other needed data. Works like a charm.

Remember reading this one just last month!