Zero-latency SQLite storage in every Durable Object | Hacker News Reader

Zero-latency SQLite storage in every Durable Object

blog.cloudflare.com

59 pointscebert11 comments

They don't mention how the Output Gates interact with things other than the HTTP response. So it seems entirely possible that you could write some data, notify some other system with a POST message, and then try to respond OK to your caller... only to lose the write. Caller doesn't get his OK, but the other system DOES receive a POSTed message. Could be a different channel, e.g. websocket. Could be a write to D1, or KV. Many ways this leads to inconsistent state.

Output gates block all output, including outgoing requests. That's why they are called "output gates" and not "response gates".

We (Cloudflare) control the whole runtime that Workers run in so it's pretty easy for us to hook every I/O mechanism in order to have it wait on the output gate.

It’d be helpful to update the blog post to make that more clear. The example focused on “constructing the response” in parallel with making the write durable, so confusing it with a response gate is an easy mistake to make.

> notify some other system with a POST message

I think this part would be blocked until the write confirmation comes back. Cloudflare can presumably achieve this in the network stack, so it should be agnostic to the code/library/platform.

Those APIs (i.e. fetch) are regular async/promise based. And in this synthetic scenario you can't construct your response-to-caller without something in that respond payload. I don't see a way to solve this all with magic.

This all strikes me as very similar to promise pipelining and distributed capabilities from an OCap system like E.

Any code sending an outbound request in reaction to a write is causally related and could be represented as a pipelined promise. The receiving system can then proceed in its work until it needs to "await" the incoming promise and can see whether it was broken due to a failure to persist some earlier write. This could also be handled at the network layer if the receiving system was external.

I'm pretty sure I remember Kenton announcing that Cloudflare Workers now supports something like (or exactly) object capabilities and promise pipelining and his knowledge/interest in such systems is already reflected in Cap'n Proto RPC.

Very cool stuff if you ask me!

You are probably thinking of Worker RPC, which is ocap-based: https://blog.cloudflare.com/javascript-native-rpc/

At present output gates operate on the scope of a single worker, blocking the output from being sent. It's easy to imagine, though, that we extend things so if you are sending a message to another worker (including a Durable Object), the message is sent immediately, but the destination worker becomes subject to the same output gate. Haven't done it yet but would definitely like to!

The part where they query individual rows through SQL queries isn't "cringe" because it is inefficient. It is "cringe", because you have degraded SQLite to nothing but an overly verbose key value store.

Shared SQL databases tend to have very poor performance when you join across shards. In this case the downside of distributed joins is avoided by having very many small purpose built databases.

> It is "cringe", because you have degraded SQLite to nothing but an overly verbose key value store.

Honestly I kind of agree with this. It feels like the main benefit over a key-value store at that point is just automatic management of secondary indexing.

That said, most users of SQLite all along have used it as a local library, and the same arguments apply to all those use cases... and yet, SQLite exists and has all these features which "seem unnecessary" in that context. So they must have value?

This looks absolutely amazing and absurd at the same time.

All IO is synchronous? Writes don't actually write when they say they do? Running 201 queries is as "fine" as running a single one with join?

It seems as they got every bad ideia in the book and somehow made it into something that works (that is, if it actually does).

Honestly really impressive. SQLite is eating the world.

Reply on news.ycombinator.com