Hacker News Reader

Comment by rockwotj | Hacker News Reader

So the reactivity here is based on smart polling?

> Later on, if any mutation inserts, updates, or deletes a record that overlaps with the read set, Convex knows it needs to recompute the listMessages query. If the result of listMessages changes, the new value is synced to the client and the component rerenders.

FWIW Firestore is able to send incremental updates and not just rerun the query when data changes. There is a complex system for broadcasting individual documents as the commit happens. I posted a bit about this a long time ago: https://news.ycombinator.com/item?id=26910411

View on HN

hi! sujay from convex here. I remember reading about your "reverse query engine" when we were getting started last year and really liking that framing of the broadcast problem here.

as james mentions, we entirely re-run the javascript function whenever we detect any of its inputs change. incrementality at this layer would be very difficult, since we're dealing with a general purpose programming language. also, since we fully sandbox and determinize these javascript "queries," the majority of the cost is in accessing the database.

eventually, I'd like to explore "reverse query execution" on the boundary between javascript and the underlying data using an approach like differential dataflow [1]. the materialize folks [2] have made a lot of progress applying it for OLAP and readyset [3] is using similar techniques for OLTP.

[1] https://github.com/TimelyDataflow/differential-dataflow

[2] https://materialize.com/

[3] https://readyset.io/

Yeah I like the model of the full function becomes the point of reactivity. It's very different than Firestore (which is trying to sync data to the device so it can run locally). It does allow Firestore to have some better offline behavior by computing queries with cached data locally, but there are benefits (which the article points out) to remote execution too.

I wish y'all the best I do think this is a super cool product!

I'm not sure what your definition of smart polling is but there is no polling going on here - the query only reruns when the data dependencies change server-side due to a subsequent mutation.

Another key distinction is that "query" here doesn't refer to a database read, it could be a complex function containing multiple reads, relatively-arbitrary compute, etc.

In the end everything is a kernel poll. Haha

Yep and every lock is a spinlock. ;-)

how does your query invalidation work internally?

We track the readset for any active subscription. When a new write transaction commits we compare the writeset for the transaction against any active subscriptions. If there's an intersection then it's likely that query will have been invalidated, so we rerun it.

I talk more about how we perform this comparison in this (now rather outdated) talk: https://youtu.be/iizcidmSwJ4?t=1218

Note that there are false-positives here, i.e., it's possible for the inputs to a query function to change without the outputs actually changing, but this is not a significant issue in practice.

Reply on news.ycombinator.com