Comment by thoughtlede | Hacker News Reader

Comment by thoughtlede | Hacker News Reader

Parent Full thread

Interesting article. Two comments:

1. I have always thought of (eventual) consistency to mean consistency between replicas: how in-sync are the replicas in what they "store". Whereas, internal consistency as defined here seems to mean how "multiple reads" can lock into the same storage state. I believe the two concepts are orthogonal, so comparing the two concepts didn't feel natural to me.

2. If transaction ids are monotonically increasing (an if), isn't it possible for subsequent reads to lock into the maximum transaction id of the first read? For example:

    select credits, max(txnid) from table;
    select debits from table where txnid <= max_txnid;

> internal consistency as defined here seems to mean how "multiple reads" can lock into the same storage state

Sort of. When you make a single read of a single key at the end of a streaming topology, that's equivalent to reading a all the inputs and doing the computation yourself. Internal consistency means that every path through the graph back to the the inputs should be reading the same set of inputs. Things go wrong when one path has processed more inputs than another, or when a single input produces multiple internal updates and those updates aren't processed atomically.

> isn't it possible for subsequent reads to lock into the maximum transaction id of the first read

Your example would still allow credits to have read more inputs than debits, but it's on the right track.

In differential dataflow aggregates wait until they see a watermark before emitting output, so the sum in `balance` will wait until the watermark for eg time 7 before emitting the sum for time 7. That allows the output from the join to settle.

Of course doing that efficiently is non-trivial...

Reply on news.ycombinator.com