Hacker News Reader

So... After following the link I am dropped in the middle of the development diary of some project. After skimming the notes I can see that the project is

a) related to data processing

b) has not yet reached the stage of practical applicability

c) is fun to hack on

And not much more. Could someone provide more context?

View on HN

A short answer: It's a recent approach to incremental computation geared towards data-science/db people. Imagine you want to maintain the connected components of a graph where edges are being added and removed. It turns out you can express this using dataflow and get reasonable looking runtimes (30,000 edge ins/del per second) [1]. You could probably do much better with a careful implementation of parallel union-find for just edge-insertions but part of the appeal of data-flow is that you express your computation using functional primitives like joins, maps and filters and then the incremental stuff 'just works'.

[1] http://www.frankmcsherry.org/differential/dataflow/2015/05/1...

this kind of thing might be interesting for local search algorithms where you want to explore different variations / adjustments of a particular structure and want to maintain some kind of topological invariants while you are doing so. E.g. search the neighbourhood of a given tree by adding / deleting edges, while preserving that what you get is in fact still a tree

Reply on news.ycombinator.com