With LiteFS, we're aiming to easily run on low resource cloud hardware such as nodes with 256MB or less of RAM. I haven't tried it on a Raspberry Pi yet but I suspect it would run fine there as well.
However, "local SQLite vs. remote Postgres/MySQL" remains a false dichotomy when talking about network latency.
I'd pick a centralized network-reachable database with a strong relational schema for write-heavy applications, and a lighter in-process system for something that is mostly reads and where latency matters. It's not a false dichotomy — but more like a continuum that certainly includes both extremes on it.
It's not like the address-space separation is without benefits... heck, if it weren't, you could simply have embedded the whole application inside Postgres and achieved the same effect.
Litestream writes everything to S3 (or similar storage).
LiteFS lets different nodes copy replicated data directly to each other over a network, without involving S3.
In either case, the actual SQLite writes and reads all happen directly against local disk, without any network traffic. Replication happens after that.
That said, unless I've misunderstood the LifeFS use case, you're still going over the network to reach a node, and that node is still going through a FUSE filesystem. That would seem to create overhead comparable (potentially more significant) to talking to a Postgres database hosted on a remote node.
It just doesn't seem that obvious that there's a big performance win here. I'd be curious to see the profiling data behind this.
Even if the reads are happening locally, if they're going through FUSE (even a "thin" pass-through that does nothing), that means they're getting routed from kernel space to a FUSE daemon, which means you're still doing IPC to another process through a kernel...
So your read queries should mostly be measured in microseconds.
You should check out the read latency for read-only requests over unix domain sockets with PostgreSQL. You tend to measure it in microseconds, and depending on circumstances it can be single-digit microseconds.
Regardless of whether your FUSE logic does nothing at all, It sure seems like there's intrinsic overhead to the FUSE model that is very similar to the intrinsic overhead of talking to another userspace database process... because you're talking to another userspace process (through the VFS layer).
When the application reads, those requests go from userspace to the FUSE driver & /dev/fuse, with the thread being put into a wait state; then the FUSE daemon needs to pull the request from /dev/fuse to service it; then your FUSE code does whatever minimal work it needs to do to process the read and passes it back through /dev/fuse and the FUSE driver, and from there back to your application. That gets you pretty much the same "block and context switch" overhead of an IPC call to Postgres (arguably more). FUSE uses splicing to minimize data copying (of course, unix domain sockets also minimize data copying), though looking at the LiteFS Go daemon, I'm not entirely sure there isn't a copy going on anyway. Memory copying issues aside, from a latency perspective, you're jumping through very similar hoops to talking to another user-space process... because that's how FUSE works.
There's a potential performance win if the data you're reading is already in the VFS cache, since that would bypass having to through the FUSE filesystem (and the /dev/fuse-to-userspace jump) entirely. The catch is, at that point you're bypassing SQLite transaction engine semantics entirely, giving you a dirty read that's really just a cached result from a previous read. That's not really a new trick, and you can get even better performance with client-side caching that can avoid a trip to kernel space.
I'm sure there's a win here somewhere, but I'm struggling to understand where.
That's a typical case for a lot of databases. Most of the "hot" data is in a small subset of the pages and many of those can live in the in-process page cache.
> The catch is, at that point you're bypassing SQLite transaction engine semantics entirely, giving you a dirty read that's really just a cached result from a previous read.
It's not bypassing the transaction engine semantics. For WAL mode, SQLite can determine when pages are updated by checking the SHM file and then reading updated pages from the WAL file. Pages in the cache don't need to flushed on every access or even between transactions to be valid.
> I'm sure there's a win here somewhere, but I'm struggling to understand where.
The main goal of LiteFS is to make it easy to globally replicate applications. Many apps run in a single region of the US (e.g. us-east-1) and that's fast for Americans but it's a 100ms round trip to Europe and a 250ms round trip to Asia. Sure, you can spin up a multi-region Postgres but it's not that easy and you'll likely deploy as separate database and application servers because Postgres is not very lightweight.
LiteFS aims to have a minimal footprint so it makes it possible to deploy many small instances since SQLite is built to run on low resource hardware.
As far as comparisons with Postgres over UNIX sockets, I agree that the performance of a single instance is probably comparable with a FUSE layer.
Yes, though most database engines end up managing their own cache and using direct IO, rather than the VFS cache.
> It's not bypassing the transaction engine semantics. For WAL mode, SQLite can determine when pages are updated by checking the SHM file and then reading updated pages from the WAL file. Pages in the cache don't need to flushed on every access or even between transactions to be valid.
That sounds a lot like at least the SHM & WAL checks wouldn't be cached by the VFS, but as I've been looking at the design more carefully, I'm starting to think I understand the idea here. Basically, the SHM & WAL get updated separately, so you might read a stale version, but since you aren't elected to be a writer, that just means you're looking at stale data, not creating an integrity problem.
> The main goal of LiteFS is to make it easy to globally replicate applications. Many apps run in a single region of the US (e.g. us-east-1) and that's fast for Americans but it's a 100ms round trip to Europe and a 250ms round trip to Asia. Sure, you can spin up a multi-region Postgres but it's not that easy and you'll likely deploy as separate database and application servers because Postgres is not very lightweight.
So, I get that multi-region Postgres can be tricky to set up, if you're doing multi-leader, but this seems about as complicated as a "single leader, many followers" set up, and given that what you're trying to do is shave off the hundreds of milliseconds from partially circumnavigating the earth at the speed of light, I'm not sure the perceived performance differences are significant (or really even measurable) compared to fluctuations in network latency of requests to the region-local node.
> LiteFS aims to have a minimal footprint so it makes it possible to deploy many small instances since SQLite is built to run on low resource hardware.
This part I'm getting and the objective a lot of sense to me (and certainly running local postgres instances on every node wouldn't be an obvious approach to me). I hadn't thought of FUSE + SQLite as a way to get there, so this is an interesting and surprising approach. I'm looking forward to how this plays out.
> As far as comparisons with Postgres over UNIX sockets, I agree that the performance of a single instance is probably comparable with a FUSE layer.
Interesting. I was thinking I was missing something. Thanks for all the insight.
1. kernel page cache fully removes the read overhead for in-memory pages
2. There's a FUSE_PASSTHROUGH mode that removes that overhead for all reads & writes. I haven't studied exactly what writes LiteFS needs to observe (just journal vs all data writes), but at least for reads it seems to pass them straight through. We could well submit a FUSE_PASSTHROUGH_READ patch to the kernel, and use that to remove all read overhead. The patch should be trivial, since the full FUSE_PASSTHROUGH mode is there already.
Disclaimer: I wrote the FUSE framework LiteFS uses, https://bazil.org/fuse
https://source.android.com/docs/core/storage/fuse-passthroug... https://lwn.net/Articles/674286/
The gains in latency with sqlite won't matter as soon as throughput starts to dominate.