Differential storage: A key building block for a DuckDB-based data warehouse
motherduck.com
motherduck.com
https://sqlite.org/cloudsqlite/doc/trunk/www/index.wiki
SQLite’s object-per-page approach means you can fetch portions of the database as needed from a bucket, whereas the DuckDB snapshotting approach seems to imply that you’d need to fetch the whole database.
Our current FUSE implementation is specific to DuckDB and does make (as well as validate) some assumptions about DuckDB's access patterns to the underlying files. In this case - we know that currently DuckDB always truncate the WAL on successful checkpoint - which triggers a Differential Storage snapshot under the hood.
We are working on future-proofing this setup by removing our reliance on some of these assumptions and having our FUSE implementation act much more like a generic FS moving forward.
Though in general the truncation of the WAL is still usually a good time to snapshot the database, as the truncation means that the only "state" needed to reconstruct the current database is completely captured in the database file.
Also, do you think any of this is going to make its way back into the duckdb core, or perhaps even influencing the duckdb developers to make some of this native or easier (avoiding assumptions about what duckdb is doing)? Perhaps some kind of trigger on checkpoint/similar activities?
And btw, very interesting to read this announcement after reading through the S3 discussion yesterday.
The EFS thing is a bit tricky. Once your credits are done it's super slow, so I bet they do provisioned but that's expensive.
The other thing was that multiple instances trying to reach the same EFS endpoint at the same time ,for the first time, will fail so you need to stagger it or retry and the docker mount plugin was a bit iffy.
That said cool concept.
Yah that's a good point. We use a similar copy-on-write implementation as many other storage systems (such as Iceberg) that offer time-travel, branching, etc... where you have snapshots represented by pointers to immutable snapshot-layer or delta files (or however else you want to call them).
In our case, we wanted to provide customers with a very similar/seamless experience of using DuckDB on MotherDuck as they would with local DuckDB. To provide this parity we needed the ability to extend DuckDB's native storage to support these capabilities (time-travel, branching, etc...). This led us to implementing Differential Storage.