w01fe

Wish I could take credit, but I think all praise (and groans) must be directed at @aria42

Thanks for the kind words, we really appreciate it! If you get a chance to check out the library, be sure to let us know what you think.

We're also anxiously awaiting this -- it seems with gvecs and reducers and primitive fns the pieces are all there, we just need the glue to put them all together. Unfortunately, for now I think we're stuck with arrays, and we're trying to make the most of it :)

We have sparse vector code built on hiphip that's slated for open-source release down the road (once we get the resources to polish it) -- stay tuned!

One of the authors here. We're excited to hear your feedback on hiphip, and will be around all day to read feedback and answer questions.

Yes, we do! Using Java from Clojure is much more pleasant than using it from Java IMO :)

Sure, analyzing code into an AST is easier in LISP (trivial, even). But you don't necessarily want to monitor every sub-function call within your function, because of performance overhead, and to limit noise. And if you want to sub out a step, the AST is not the most natural data structure to work with.

Graph forces you to make the steps that you care about explicit, and in exchange you get a nice way to observe, reason about, and change your code in terms of these steps. The goal is to make the overall process as clear and non-magical as possible, while incurring as little programmer overhead as possible.

I think it's a really interesting project to attempt to provide similar tools over ordinary functions, but that seems like a much loftier goal -- Graph is pragmatic, simple, and it works now :).

Short answer: because Graphs are data, it's easy to do tons of things with them that are difficult to do with code. In principle tooling may eventually bridge the gap, but for now it's hard to take a function and automatically monitor it's sub-functions, or run up to a particular intermediate result, or substitute one step for another in a test, etc. Our previous blog post gives some more detailed examples:

http://blog.getprismatic.com/blog/2012/10/1/prismatics-graph...

Re: interactive visualizations, we haven't gotten there yet, but it sounds like a really cool (and feasible) idea. On this front, libraries like 'lamina' look like a nice place to start.

https://github.com/ztellman/lamina

Graph backtraces look like ordinary stacktraces -- the compiled output is basically the same if you wrote the function by hand.

For logging, we wrap each node in an 'observer' with the path through the graph injected, which automatically records execution time and exceptions from each node, and lets you spit stuff out to the dashboard that will appear in the graph structure. There's an example of this in the graph_examples_test.clj, I believe.

Graph is currently just in-process (no cross-machine distribution), although it's definitely a possibility down the line. The currently released parallel strategy is also just a pedagogical example, not really meant to be used.

Here's some WIP on a more practical parallel compilation: https://gist.github.com/w01fe/4710008 Once the kinks get worked out this will go into the OSS project. Presumably concurrency level will be controlled by a parameter, and/or passing an appropriate ExecutorService.

Coauthor of Graph here, I'm happy to answer any questions.

We're working on it! We removed the software because it needed extensive cross-project reorganization, consolidation, and cleanup which just wasn't possible to do with a clean upgrade path using available resources (3 backend engineers). Plans are to re-release all of this and more, and we've already started with Plumbing and Graph: https://github.com/prismatic/plumbing. Which librar(ies) in particular are you most interested in?

Fine-grained oauth is clearly better and something we plan to do eventually, but we're a very small team and we've been prioritizing building features and improving the design and relevance for the time being.

I understand your concern ... see my response here:

https://news.ycombinator.com/item?id=5373701

Short answer, we don't want access to manage your contacts, there's just not a read-only option (last time I checked).

We ask to read your contacts to autocomplete email addresses for email shares and help you find friends on Prismatic (if you choose to do so). Last time I checked, Google didn't have a read-only contacts permission level, or we'd be using it.

Thanks for the feedback -- this is a great idea, and something we've been thinking about for awhile ... stay tuned :)

Don't worry, we're on your side :) We ask to read your contacts to help autocomplete email addresses for email shares and help you find friends on Prismatic (if you choose to do so). Last time I checked, Google didn't have a read-only contacts permission level, or we'd be using it.

Sorry for your bad experience. The import adds both your subscriptions as well as topic feeds we think you'll be interested in based on your GR activity, so it's not going to be exactly the same content. You can remove them if you like by going from 'interests' in the home or profile header.

We currently process delete account requests manually -- if you use the feedback tool within the app or email us at feedback@getprismatic.com we'll be happy to take care of it for you. Adding this as an option directly within the app is in the works as well.

Right on all counts. On the upside, we're smarter than an RSS reader for high-volume feeds currently, and sort things based on how much we think you'll like them (based on topics, social information, and more) rather than just by time. Support for the 100%-of-feed use case is in the works, stay tuned.

I'd love to chat about this as well -- I'll send you an email.

Interesting, I hadn't heard of StructureMap. It seems related, but Graph is less complex -- just the dependency and composition parts, without being tied to any particular use case.

Any particular execution is fixed once it's compiled. But it's easy to compile different variants of a graph and choose between them based on the input parameters, if that's all you need.

I see. For streaming computations we typically have a Graph behind a thread pool, so a node can always resubmit a datum for another go-round -- there's no concept of sending an updated datum 'back' to another node within a particular execution though.

Interesting! What do you mean by 'bookkeeping'?

> fnk's use their argument names to define how to connect > edges of the graph together. This means that fnk's are not > modular

> You couldn't make a utility library of commonly used fnk's > because you would need their argument names to match up > with the graph you are using.

This is a great point. This is an issue that we've largely solved in our own codebase, but haven't quite polished yet -- look for a release soon.

Long-story short, we have a macro 'instance' that works on fnks and graphs, which looks like this:

(def graph-using-stats (graph :my-data ... :stats (instance stats-graph [my-data] {:xs my-data})))

For the fnk case, instance can be defined just as:

(defmacro instance ([f bind m] `(pfnk/comp-partial ~f (fnk ~bind ~m))))

This allows you to provide arguments to a subgraph or node fnk via arbitrary computations on input parameters or other node values, including the trivial case of renaming.

With this in place, you can always name your fnk arguments and Graph nodes whatever makes sense in this particular context, and then adapt the graph to a new circumstance using instance.

We use this strategy extensively across our codebase, and will provide lots more examples as we release more of our infrastructure. Please let me know if this makes sense, seems reasonable to you, or you have questions.

> If no graph node is hooked up to the 1TB of random number > output, the Graph compiler should optimize it out and never > run it. Is that possible with sub-graphs?

Yes, one of the design goals of Graphs is to make everything transparent, until the last second when you compile a Graph. Our current compilation strategies are pretty simple (and it's very simple to build your own), but right now you can lazily compile a hierarchical graph and any results that are unused (including in subgraphs) will not be executed.

Thanks!

If by 'handle cycles', you mean 'throw an exception', then yes :). Graph models single-pass data flows, which must be acyclic, and the (graph) constructer and (*-compile) methods throw if you give them cyclic specifications. Do you have a particular use case in mind where cycles are desirable?

For now we're focusing on the in-process use case, which we think is underserved and allows the simplicity of Graph to really shine. That said, distributed Graphs (and possibly, integration with frameworks like Storm) are on the horizon. If this is something you're interested in working with us on, please let us know.

Thanks! We'd love to hear your feedback -- and if Graph doesn't meet your needs, work with you to fix that.

Also, here's a direct link to the source:

https://github.com/prismatic/plumbing

and a literate test with lots of real examples:

https://github.com/Prismatic/plumbing/blob/master/test/plumb...

We're also working on other kinds of compilation, including 'direct' ones that compile directly to a single fn, and 'async' ones that handle asynchronous functions and are smarter about spinning up threads:

https://gist.github.com/w01fe/4710008

I programmed in CL for several years exclusively, and think it's an awesome language. But I also really love Clojure, and think it's the the most beautiful Lisp (or S-expression-oriented language with a read-eval-print loop, if you prefer) I've had the opportunity to explore. To each their own.