Hacker News Reader

Comment by frankmcsherry | Hacker News Reader

For some context on "what Niko presented at the Conj",

    https://2018.clojure-conj.org/nikolas-gobel/

which is a reactive Datalog implementation built over differential dataflow. The .. um reported anxiety that to add new queries requires a recompile loop, replaying all of history, etc, suggests maybe they aren't entirely on top of how the system works.

Edit: and again tomorrow at clojureD:

    https://clojured.de/schedule/#nikolasGoebel

View on HN

hi Frank, we are definitely not entirely on top of this stuff, but it is very awesome and we will get there eventually :)

Can you help me understand where we went awry here:

"Now of course the problem there with a rules engine... I have to look at the incremental dataflow, but I imagine it's kind of the same sort of approach. The problem with that is that if you want to sort by a new thing or you have a new piece of information, you have to change the network and backflow any... [You have to know compile time...] Yeah. Yeah. Exactly. And there are systems where you can add a new rule at runtime, but a lot of those also involve then re-ingesting history into the network to repopulate anything they... 'Cause a lot of these systems to save memory, will just throw away any information you're not interested in, from compiled query."

If we are close can you tighten up the language and if we are far where did we go wrong? (In the context of differential dataflow, not rules networks)

No worries!

Niko's talk should probably be the reference material, but you definitely can deploy new queries interactively, against maintained indexed representations of the historical streams. No recompilation required. Streams are maintained in indexed form and don't need to be re-flowed (they may need to be re-indexed if your query calls for an index that doesn't exist).

Deploying a new query takes ~milliseconds if the correct indices exist, and if not it seems to be no slower than purpose-built Datalog systems (e.g. LogicBlox, but performance seems solid compared with Souffle). This is despite then running against live-updated data.

As I suspect you know, we cannot report performance relative to Datomic. ;)

Reply on news.ycombinator.com