> we tail postgres’ WAL to detect novelty and use last-write-win semantics to handle conflicts
can you elaborate more on how you achieve this
can you elaborate more on how you achieve this
For example, given a query like "fetch user where id = 1", there could be a topic like "users:id:1" [1]
When a WAL record comes in, we find queries by their topics, and invalidate them. This triggers a refresh.
This is inspired by Figma's LiveGraph [2], which in turn is inspired by Asana's Luna [3]. The essays cover the idea in more detail, but you can also start diving into the code, at invalidator.clj [4]
[1] This is an example. Inside Instant, we encode them as a 'datalog' pattern. Something like `[:ea "1" :users/id]` [2] https://blog.asana.com/2020/09/worldstore-distributed-cachin... [3] https://www.figma.com/blog/livegraph-real-time-data-fetching... [4] https://github.com/instantdb/instant/blob/main/server/src/in...
1. first we would use the WAL records to invalidate queries that could be affected (with optimizations for fast matching) and requery the data
2. then we used the info from the WAL record to update the query in-memory without asking the DB for the new result, it worked for majority of the queries that can be reliably modeled outside of the DB
3. I believe after I left the team reverted to the re-query approach, as managing a system that replicates the DB behavior was not something they were excited to maintain, and as the DB layer got scaled out, extra DB queries were less of a downside
I'll reach out to you on twitter; would love to learn more about your experience