Introducing ReadySet

I've used both of the suggested methods under "Current standards for scaling out databases" so I see where this is coming from. But I peeked at the AWS reference architecture, and it places a Consul and ReadySet deployment in my environment for me to run and maintain. I feel like any sales pitch for this really needs to convince me that having these things in my environments is going to be worth the hassle in terms of milliseconds and dollars, as opposed to just using RDS read replicas and paying a bit more. Then again, I can see this being an obvious choice if you're growing very quickly or have tight latency requirements.

With that said, it looks like cool tech and I read Jon's Rust for Rustaceans which serves as a stamp of quality for this even if I haven't tried it yet!

Digging down a couple layers of links from this, the underlying paper, "Partial State in Dataflow-Based Materialized Views" https://jon.thesquareplanet.com/papers/phd-thesis.pdf is pretty intriguing. It sounds like a potential free lunch in specific performance areas, which means it also sounds too good to be true, but if it turns out to be a metaphorical 90%-off lunch that's still very promising.

I am really excited to see where readyset can take this technology, if deployment is as simple as they say this sounds like an instant win for any high throughput service.

I am curious how they handle queries that would overflow local main memory, like if I just had a PK lookup on a 10TB table you obviously can't store all that in RAM, and would still need to do some form of cache invalidation.

This was recently on my radar for a thing i wanted to do, but what you have might also fill the same gap.

https://materialize.com/

I can't think of real-world examples of apps that have read path scaling problem that this tech would solve well. It would be great if the authors can catalogue a bunch of real-world use-cases. (real-world customer case-studies would be even better).

Today, machines are super huge (in terms of compute cores, memory and iops for storage and network) and a single Mysql or PostgreSQL database can do a lot of work. This makes it much much easier to build apps that don't have as many users at Internet scale – that is pretty much all enterprise apps – without resorting to distributed databases.

In Internet scale consumer app domains like e-commerce/delivery or fintech where relational databases are used heavily, most queries would have strict correctness requirements and won't tolerate staleness. Also, most query-results would be highly specific to each user and won't have much cache hits. Also, apps in general are increasingly personalised and have fast changing content.

In terms of technology evolution, I see people moving from single large machine databases to distributed sql databases as their use-cases scale.

And as distributed sql databases mature, I expect they will get built-in capability to generate user-defined materialised views with flexibility to manage their placement w.r.t class and number of machines to compute and serve them etc.

This seems like it could be a good fit for my team (and we have been discussing this kind of caching).

We frequently (~ once per second) run queries over relations that are increasing at a rate of ~100 rows per second (append only, no updates).

Could this cause any performance concerns for ReadySet? How much control do we have over the frequency of reconstruction of cached data based on the flow graph?

This feels like the sort of the where it works great 90% of the time, but as soon as you want to do something more complicated/nontrivial it doesn’t handle it properly and is impossible to debug/improve since you are using an opaque solution.

At least with scaling replicas or having a dumb cache layer it’s easy to understand the system.

I was prepared to be super skeptical here, but this actually looks like it could be really good, without all the gotchas I was expecting. I think the only potential hole I see is if you do a lot of db-side code as stored procedures and what-not. I can’t tell from the write up if they can keep those consistent.

Hi, Jon Gjengset I am so happy to see attempt to make Noria usable in legacy applications. Am I right in assuming this need to consume the replication log from the primary database? Or write request going through ReadSet proxy will produce its own change feed ?

Oh this is like litestream that featured in TailScale's post "A database for 2022" discussed recently [1]. In both cases you're replicating by tailing the database WAL, in this case Postgresql/MySql and in litestream's case SQLite. It's a great idea, I'm glad we're starting to take advantage of this great opportunity afforded by the design convergence of using a WAL for database transactions.

[1]: https://news.ycombinator.com/item?id=30883015

I am wondering how the financial dividends from this are set up. Obviously Jon wrote the paper and core technology behind it, and I am hoping he can do well from it.

What are the biggest difference with something like materialize.com?

I love the way you explain what's ReadySet. Congrats on the launch.

> However, if your million-dollar idea ends up being worth even $100K, you’ll likely find your database struggling to keep up with the scale.

Huh? At a 3 year multiple that is less than $3,000 monthly revenue. I'm sure a SaaS can provide way more value for $3k/mo before you start running into scaling issues

Lots of database proxies and caches like ProxySQL or Heimdall Data or IMDG's like Apache Ignite - but it's interesting to see it offered as a cloud service backed with this cache invalidation tech.

Had a similar idea a few years back and it's nice to see it turn real. Congrats on the launch.

Very cool tech here, excited for the future. Congrats on fundraising!

Is it open source?

Sounds great, until you realize you are switching to eventual consistency for your whole application.

I am wondering if one could setup ReadySet on top of Vitness / PlanetScale.

Could you explain the GDPR use case? That seems more like the use case for data residency and not data retrieval?

Can this compile to WebAssembly and run in the browser, in front of SQLite?