Hacker News Reader

Comment by dkhenry | Hacker News Reader

I am really excited to see where readyset can take this technology, if deployment is as simple as they say this sounds like an instant win for any high throughput service.

I am curious how they handle queries that would overflow local main memory, like if I just had a PK lookup on a 10TB table you obviously can't store all that in RAM, and would still need to do some form of cache invalidation.

View on HN

The trick is "partial view materialization" (https://jon.thesquareplanet.com/papers/phd-thesis.pdf). Basically, you only materialize results for commonly-accessed keys, and compute other keys on-demand.

Is there a way to federate which keys are commonly accessed ? Like if I commonly access the entire table can I direct inbound traffic to different application servers and have them access different caches so each cache can pull only a subset of the data into the cache, and not worry about things like which keys are being written globally

We've thought about that, actually! We have an experimental mode where multiple copies of the same query can be created (actually just multiple copies of the leaf node in the dataflow graph, so intermediate state is reused) with different subsets of keys materialized - the idea is then that these separate readers would be run on different regions, so eg the reader in the EU region gets keys for EU users, and the reader in the NA region gets keys for NA users.

Reply on news.ycombinator.com