Hacker News Reader

Comment by sedev | Hacker News Reader

Digging down a couple layers of links from this, the underlying paper, "Partial State in Dataflow-Based Materialized Views" https://jon.thesquareplanet.com/papers/phd-thesis.pdf is pretty intriguing. It sounds like a potential free lunch in specific performance areas, which means it also sounds too good to be true, but if it turns out to be a metaphorical 90%-off lunch that's still very promising.

View on HN

Oh hey, that's my thesis! Happy to answer any questions you may have about it :) There's also the OSDI'18 paper here which may be of interest: https://jon.tsp.io/papers/osdi18-noria.pdf

I haven't read much about Noria other than this readme [0], but would like to know if you are familiar enough to contrast Materialize [1] with it in terms of perf, overhead, approach, and fundamental (design) principles?

[0] https://github.com/mit-pdos/noria

[1] https://github.com/MaterializeInc/materialize

Back when we announced Materialize we got the same question in reverse! You can read my response from a few years back here: https://news.ycombinator.com/item?id=22362301

Unfortunately I'm not privy to whatever improvements ReadySet has made in the past two years, so I can't comment on differences between ReadySet and Materialize. Perhaps Jon can, though!

Insightful. Thanks!

This is super exciting. Ever since I talked to you about Noria I've been telling people about this concept. I'm excited to see a production ready implementation of it.

Big fan here!

I've been following the space since a bit of time, and I must say it's exciting. To me this is the future of apps where the Truth lives server-side, and everything reacts from there; With partial state evaluation lowering resource consumption to a minimum.

Kafka Streams and Apache Flink seem to be focused on real-time analytics, and I wish they'd get there to stimulate the space.

Are you affiliated with ReadySet?

I'm pretty excited about it too! I remember when I initially started the research I was amazed that this didn't already exist.

Some context: https://twitter.com/jonhoo/status/1511401461669720068

Basically, I co-founded the company around the time I graduated, but had had my fill of database research after six years of PhD. So I joined AWS to work on Rust while Alana (the CEO) took on leading ReadySet.

According to the linked article, Jon appears to be one of the co-founders

the remaining 10% of the 90%-off free lunch is pretty much just eventual consistency - it can occasionally be the case that you write something to the DB, and an immediate subsequent write doesn't see it. That said, there are escape-hatches there (we'll proxy queries that happen inside of a transaction to the upstream mysql/postgresql database, and there's an experimental implementation of opt-in Read-Your-Writes consistency), and I'd wager that the vast majority of "traditional" web applications can tolerate slightly stale reads.

Our official docs also have an aptly-titled: "what's the catch?" section: https://docs.readyset.io/concepts/overview#whats-the-catch

this is similar to an asynchronously replicated database replica.

Which mean read might return stale data because of the replication lag.

It will also increase the load on the server you read the replication log from.

But the primary database could dump the transaction log to (S3/kafka) and have ReadySet instance read it from there instead of directly from Primary database.

So for a read mostly website (hackernews/reddit) this is indeed a free lunch.

Reply on news.ycombinator.com