Hacker News Reader

Comment by w01fe | Hacker News Reader

Coauthor of Graph here, I'm happy to answer any questions.

It's not clear to me how Graph is better than just regular pure functional programming. In theory, you could implement graph/par-compile for regular functions by examining the dependence graph in their code, right?

Is the special-purpose declarative syntax useful in its own right?

Short answer: because Graphs are data, it's easy to do tons of things with them that are difficult to do with code. In principle tooling may eventually bridge the gap, but for now it's hard to take a function and automatically monitor it's sub-functions, or run up to a particular intermediate result, or substitute one step for another in a test, etc. Our previous blog post gives some more detailed examples:

http://blog.getprismatic.com/blog/2012/10/1/prismatics-graph...

You're using LISP, so code is data, right? And if all your functions are pure, the code should have a nice dependence-graph structure.

I can believe that the actual LISP data structure of a function may not be the most convenient to work with for what you're doing, but it seems like you ought to be able to translate from LISP code into whatever graph structure you want, as long as all the function calls are pure. Or are there reasons this isn't feasible?

Sure, analyzing code into an AST is easier in LISP (trivial, even). But you don't necessarily want to monitor every sub-function call within your function, because of performance overhead, and to limit noise. And if you want to sub out a step, the AST is not the most natural data structure to work with.

Graph forces you to make the steps that you care about explicit, and in exchange you get a nice way to observe, reason about, and change your code in terms of these steps. The goal is to make the overall process as clear and non-magical as possible, while incurring as little programmer overhead as possible.

I think it's a really interesting project to attempt to provide similar tools over ordinary functions, but that seems like a much loftier goal -- Graph is pragmatic, simple, and it works now :).

Beyond using graphviz on your topology, can you interactively visualize the results flowing along each edge? What does a "graph backtrace" look like? How do you do logging -- does something like Zipkin help?

I feel like we still don't know what more traditional tools and workflows look like from the graph point-of-view.

Edit: cool project, btw.

Re: interactive visualizations, we haven't gotten there yet, but it sounds like a really cool (and feasible) idea. On this front, libraries like 'lamina' look like a nice place to start.

https://github.com/ztellman/lamina

Graph backtraces look like ordinary stacktraces -- the compiled output is basically the same if you wrote the function by hand.

For logging, we wrap each node in an 'observer' with the path through the graph injected, which automatically records execution time and exceptions from each node, and lets you spit stuff out to the dashboard that will appear in the graph structure. There's an example of this in the graph_examples_test.clj, I believe.

How does the parallel "execution strategy" work? Do you specify a maximum concurrency level? Can you distribute the computation onto other machines?

Graph is currently just in-process (no cross-machine distribution), although it's definitely a possibility down the line. The currently released parallel strategy is also just a pedagogical example, not really meant to be used.

Here's some WIP on a more practical parallel compilation: https://gist.github.com/w01fe/4710008 Once the kinks get worked out this will go into the OSS project. Presumably concurrency level will be controlled by a parameter, and/or passing an appropriate ExecutorService.

It would be very interesting if you could easily convert a graph into a Storm[0] topology. An example of the Storm Clojure DSL is available in the storm-starter project[1].

[0] http://storm-project.net/

[1] https://github.com/nathanmarz/storm-starter/blob/master/src/...

I'm not sure how it's implemented in Graph, but the best thing you can do is to abstract out an execution to a general object.

From there, you can create all sorts of adapters and the "graph" engine just farms out the execution to the appropriate adapter. This could be anything from a job daemon on a batch farm, or, say, a typed ThreadPoolExecutor on a machine (which can either execute native clojure/java, or any sort of other script language.

This would be brilliant. Does any other framework support this? (outside Clojure)

Reply on news.ycombinator.com