The Session Extension
sqlite.org
sqlite.org
SQLite never ceases to amaze me.
Using Syncthing as a central storage for NNCP packets circumvents the strict routing requirement.
I don’t think it’ll be useful here
> When processing an UPDATE change, the following conflicts may be detected:
> The target database may contain a row with the specified PRIMARY KEY values, but the current values of the fields that will be modified by the change may not match the original values stored within the changeset. This type of conflict is not detected when using a patchset.
Edit:
I wander if there is a way to add a last_mod column to the tables and so only a more recent UPDATE would be executed from a patch set? It would only work at the row level but for most cases that should be ok. You could always create other tables for more granular control.
It looks like you can provide a conflict hander too that can replace the update statement, that could be useful for merging conflicting changes.
This could be exactly what I want for something!
Clock skew is an insidious problem: things may look like they’re working, but they’re not. Given all of the clock bugs out there it’s sometimes not even safe to rely on the clock on a single machine. Across multiple machines it’s hopeless.
Please note that this doesn’t apply if you have a fancy time source like Google uses in their fancy cloud DB (whose name escapes me at the moment).
I’m looking for a system using SQLite that gives you the same eventual consistency guarantees as CouchDB/PouchDB on consumer devices. You need to be able to have nodes disconnect from each other and the internet for long periods of time before syncing. Hours or days not seconds or milliseconds.
If two nodes overwrite the same row while disconnected you need a deterministic way to decide which is the winner. This could be though a hash of the values in the row (lowest value wins) or using a timestamp. Pros and cons for each and it depends on use case. So for example if you are building a notebook application you almost certainly want the most recent update to win. The transaction rate would be low and devices with an internet connection will be “relatively” accurate on time.
Obviously this could be open to abuse, someone changing their clock to force their edit to win. But again pros and cons. The use case I’m looking for is within a trusted group where this wouldn’t be a problem.
The use case is an upgrade from postgres 10 to 11. The last time we upgraded (to 13), some queries turned out not to work well, so we were forced to go back to the old server. In the meantime stuff was written to the new server which we then manually put into the old db before switching over to the old. Result: lots of downtime.
Is there a way to create a changeset like this in postgres?
See: Situations Where A Client/Server RDBMS May Work Better
As much as I love the idea, Raft is not a silver bullet and can break in strange and inconspicuous ways that may be difficult to fix. It can also lead to subtle inconsistencies in the data depending on what data you are putting in it. Overall it may be good in some situations, but I doubt it will dislodge Postgres in any meaningful way.
Edit: from rqlite's FAQ
> Raft is a Consistency-Partition (CP) protocol. This means that if a rqlite cluster is partitioned, only the side of the cluster that contains a majority of the nodes will be available.
Disconcertingly, this sounds like they assume that only one side of a partition can contain a majority of nodes. This isn't true if the partition is partial, eg A<-->B, B<-->C, A<-/->C.
rqlite (and Raft) doesn't assume this. All it states is that for the cluster to make progress i.e. apply a change in a consistent manner, at least a quorum of nodes must be online and in contact with the Leader (which is one of the online nodes). Every node in a rqlite cluster knows what size the cluster is and therefore requires that (N/2)+1 nodes acknowledge the change before that change is committed. The Leader node performs this coordination.
The scenario you outlined above is obviously possible. But in that event the cluster is down -- the Leader (let's say it's node A) cannot contact a quorum of nodes. No changes can be made to it.
In my experience, any inconsistencies would be due to misuse of Raft, not something intrinsic to Raft itself.
I was referring to the issues around determinism in queries. These are easy enough to catch if you’re aware of them, but it pushes more onus onto the application layer / devs (not that it’s necessarily a bad thing, just a trade-off).
The comment you're replying to is saying rather more than that, and I don't endorse it, but the shift in the possibility space is still both fascinating and impressive.
(also I love sqlite's tendency to use pg as a syntax template, it seems like an example of two projects with significantly different goals playing to each others' strengths in a beautifully positive sum game fashion)
If you take a sharding approach to concurrency, you might just be happy with no write concurrency: one process/thread per-CPU / storage slice, and fix everything else upstairs. It's not a crazy approach, and it might be the approach many would take if they could start from scratch. In fact, it's the approach Spanner takes, for example.
As others point out in this thread, there's already rqlite for Go, sqlite for C, and now we've open sourced our work on bringing SQLite and Raft to Rust: https://glaubercosta-11125.medium.com/winds-of-change-in-web...