Curious if you prototyped DuckDB before deciding to invest further into SQLite.
DuckDB works great as an in-memory database (it's also the default mode).
DuckDB works great as an in-memory database (it's also the default mode).
I'm sure there's a lot of really cool local-first databases out there, but SQLite has the benefit of being incredibly widely battle-tested, with literally billions of installations worldwide. It has received thorough security research and fuzzing (it's part of Chrome's attack surface after all). And there's tons of resources online to help people understand how to use it. Although I'm sure there are alternatives that serve certain use cases better it's hard to imagine anything coming close for ours.
That said, the storage engine we've built is not that heavily dependent on SQLite specifically. Any database that uses a write-ahead log like SQLite does should be possible to adapt to it in the future. So maybe we'll eventually open it up to a variety of choices, or even let you bring your own as a Wasm module.
> DuckDB is indeed a free columnar database system, but it is not entirely built on top of SQLite. It exposes the same front-end and uses components of SQLite (the shell and testing infrastructure), but the execution engine/storage code is new.
(I use it personally, but it's not the same thing as what we're building with D1)
Basically, we already have a product for doing DuckDB-like things, and the D1 architecture isn't great for high volume OLAP workloads.
Disclaimer: I'm an engineer at Cloudflare, but not on a developer platform team. I'm not speaking for our developer platform strategy or anything, I'm just commenting on how it looks from where I sit.