Litestream live replication has been moved to the LiteFS project
github.com
github.com
> > To improve latency, we're aiming at a scale-out model that works similarly to Fly Postgres. That's to say: writes get forwarded to the primary and all read requests get served from their local copies.
> How can you ensure that a client that just performed a forwarded write will be able to read that back on their local replica on subsequent reads?
> LiteFS provides a transaction ID that applications can use to determine replication lag. If the replica is behind the TXID, it can either wait or it can forward to the primary to ensure consistency.
[1]: https://news.ycombinator.com/item?id=32925734#32928974
[2]: I think this is a reasonable statement, but may not be industry standard terminology.
https://nats.io/about/: "[NATS] enables applications to securely communicate across any combination of cloud vendors, on-premise, edge, web and mobile, and devices. […] The NATS Server acts as a central nervous system for building distributed applications."
FYI to anyone here, FoundationDB is fucking awesome for something like this.
Question @losfair: Did you find the Rust bindings for FDB to be very good? The Go bindings are OK, but are pretty out-of-date with some cool new features on the HEAD of the FDB source repo.
I imagine you’d get an FDB transaction time limit error preventing any schema migrations with non trivial amounts of data.
The entire SQLite journaling mechanism is not used by mvSQLite (you can set journal_mode=off safely - although SQLite won't be happy to do explicit rollback in this case)
SQLite is unfortunately (?) kind of hard to modify for external processes and while it's built very extensibly it's often not the thing you can kind of just... turn on, if that makes sense, and SQLite lives in the address space of the executing program.
You end up with stuff like hacking LD_PRELOAD[0].
Note: Litestream (Ben) was acquihired essentially by Fly.io (so that should explain all their recent SQLite content!).
However, I disagree that it's just a "sequence of experiments". There are a lot of applications that can benefit from asynchronous replication. Synchronous acknowledgement of writes can impose a high cost on throughput so many folks use async replication, even in systems like Postgres.
- rare writes that get directed to a single instance (e.g. using Fly.io's replay header), frequent reads (potentially at edge locations)
- no need to deploy a PostgreSQL cluster and set up logical replication
- SQLite database stored in object storage, reader replicas can boot up using the object storage copy and then get kept in sync by pulling data from the writer
- delay in replication is fine
LiteFS is probably going to be a great solution here since we're mainly using Fly.io and it has built-in support for it [1], but are there any alternatives that don't require Consul, still look like an SQLite database to the client and can work off of a HTTP connection to the primary, so that we don't have to require our users to deploy to Fly?
The way Litestream was doing replication required programmers to be extremely careful not to accidentally attempt a write to a replicated database copy - doing so would corrupt that copy, potentially in non-obvious ways. There was no practical mechanism for protecting people from making this mistake.
The FUSE approach for LiteFS has more control, so can do a better job of protecting people from this kind of mistake.
Another benefit of the control is that we can maintain a rolling checksum of the entire database at each transaction so we're able to verify integrity when replicating. That's also what allows us to do asynchronous replication across a loose membership of nodes since we can easily detect if a node diverges.
You have to write to the SQLite via the rqlite HTTP API but it will replicate the data to N nodes (at least 20) via RAFT and then others can read-only the SQLite replica files directly; and file-permissions prevent the accidental write.
> You have to write to the SQLite via the rqlite HTTP API
Also requiring non determinism (i.e. no RANDOM()) is something I don't think I really want to worry about. There are a few of tradeoffs for rqlite (and dqlite too, to be fair) that just don't seem to be quite worth it (especially compared to just running Postgres).
I think people are realizing that having one far away writer is actually fine -- 90%+ of the traffic you're trying to serve fast is read queries.
[1] https://www.philipotoole.com/rqlite-7-7-0-released/
[2] https://github.com/rqlite/rqlite/blob/master/DOC/NON_DETERMI...
Requirements and features change with time, don't fret about names too much.
Is there an issue with OS compatibility? FUSE tends to require OS hooks, last I checked, and that can be somewhat hairy to deal with.
It's a tricky question to answer. Most of the noticeable overhead is on the write side. Initial benchmarks of overhead that I've seen locally are about 250µs for the write(2) and fsync(2) calls. It's closer to 100µs for read(2) calls. There are additional writes made behind the scenes as well for storing in a replication format for the other nodes too.
However, on the read side some of that is moot. For many databases, most reads will be in the OS page cache and a fetch from there seems to be closer to 4µs. If you're running a moderately sized database (e.g 1GB) on even a modest VM (e.g. 256 RAM) then most of your hot pages will be in the OS page cache so you shouldn't notice much overhead on the read side.
LiteFS is targeted at read-heavy workloads. If you need high write throughput of thousands of writes per second then LiteFS probably isn't a good fit.
> Is there an issue with OS compatibility?
LiteFS is Linux right now. We'll be supporting other operating systems via a SQLite VFS extension in the future. macOS has poor FUSE support right now and I'm not sure where Windows and BSD stand with their support for FUSE or FUSE-like systems.
Are there advantages to a FUSE-based approach over a VFS-based approach?
For the CLI that's a fair concern, but from an "I'm writing an application that links against libsqlite3 and/or includes sqlite3.c" I feel like the opposite is true: I'll have a much easier time configuring the VFS in my app than fiddling with FUSE mounts.
> LiteFS runs an HTTP API server to communicate between nodes as well and if you run multiple processes with a VFS then you have to determine which one should start up that server.
Wouldn't that be a similar problem as working out which process should be the writer? I'm admittedly not too familiar with either Litestream or LiteFS, so that might be a dumb question.
[1]: https://github.com/superfly/litefs/issues/119#issuecomment-1...
just as no one using postgres on RDS will ever leave RDS not b/c RDS is so much better than its competitors but because the hurdle is too great and the migration so risky. right now, fly is the only one who lessens the burden to use LiteFS and as long as they're the only one, the average developer is essentially locked in.