This is super interesting @richieartoul. I specced out something similar myself and was going to implement it in Zig https://github.com/fremantle-industries/transit.

FWIW I came to a similar conclusion that a lot of the power in Kafka comes from the API and that eventually much of the complexity of managing the cluster will eventually be abstracted away with multiple implementations. I also felt that if I could implement Kafka persistent over the S3 keyspace then I could start with persistence direct to S3 like you've done with warpstream and then layer on a faster hot disk and in memory tiering mechanism to eventually lower end to end latencies.

I love where you're going with this so hit me up on twitter if you ever want to chat more in-depth https://twitter.com/rupurt.