Prismatic's "Graph" at Strange Loop | Hacker News Reader

Prismatic's "Graph" at Strange Loop

blog.getprismatic.com

106 pointsjanaboruta45 comments

I love reading prismatic's blog. You have some really smart people there. However, it seems like you're trying to solve everything from a purist view and I think your product is suffering as a result. I have made the same suggestion consistently that would make me use it more without any reply. Basically, I've stopped using the product as the UX does not match the reason I would want to use the product.

I'm sorry about that. If you're still interested in giving us UX feedback send us an email feedback@getprismatic.com. I'll personally respond to it. Please understand that we're still a small team and we get a lot of emails on a daily basis. :-)

I agree with the author, the UX is definitely under-developed and I've also given many feedback msgs.

We only say this because we care :)

I'm curious what your complaints are, do you mind sharing? Especially because you invoke "purist view", which is rather confusing talking about UX, a very subjetive area.

For myself, I feel that the main feed could use a lot more whitespace and have less information about each story. For example, I wouldn't mind being able to toggle off the social quotes completely and perhaps even the images (images are nice for some sources, but not others).

This sounds very useful. I'm having trouble understanding the nature of the system, though, because I don't really understand lisp / clojure. Is this a pattern that could be applied in any language with first class functions, or does it require a lisp to be practical?

Sure! This particular way of expressing declarative structure may be specific to Clojure, but the basic ideas should be useful in any (especially dynamic) language with first-class functions.

The first idea is to take a complex system and express the set of components and their relationships declaratively, rather than procedurally. This is an old idea, and makes it possible to reason about the system, mock out components for testing, map over the components for monitoring, and more.

The second idea is simplicity. By making this declarative language as simple as possible, it becomes trivially easy to do all the things I just mentioned using the existing tools in the language, without needing to write complex library functions to support each use case.

For another example of a declarative system for composition you could look at 'react' in JS, which can be used in similar ways to Graph (and also supports things like async composition): https://github.com/jeffbski/react

react looks interesting, although man, the stringly-typed bit sorta reinforces a lot of the Lisp quotes about being able to modify the language, etc. (Disclaimer: I've only just started studying Clojure in earnest over the last month or two. No zealot like a convert, eh?)

Thanks! That makes a lot of sense, and react looks like a really interesting project. I'll have to go back and reread the article with an eye for applying its ideas in Javascript / Coffeescript.

You can do it in any language with first class functions and reflection.

Here's a stupid simple js implementation of their example:

https://gist.github.com/3874826

Edit: Loaded the page and got interrupted by lunch. Shows me for not refreshing.

That's awesome -- thanks for posting

Thanks for writing it up. I've thought about dataflow programming before but building the dependency graph up was always annoying and the idea of sticking everything in a map and building the dep chains by introspecting the argument names hadn't occurred to me. With the ease of use solved this is handy enough that I'll probably turn it into a real promise-based utility.

Awesome, I'm excited to see what you come up with. If you do end up making something, please email me a link (I wrote you earlier) since I'm sure there's lots to be improved in our implementation.

Pretty cool project, like a functional reinterpretation of dependency injection frameworks. The basic idea behind dependency injection is to make all dependencies for each unit explicit and externally supplied.

DI frameworks then build upon the pattern offering a declarative DSL to declare dependencies and wiring. They also offer a selection of alternatives to build dependent objects (e.g. build a fresh one when needed or supply a global instance for every dependee) and a smidgen of generic functionality via impersonation (e.g. monitoring, logging, timing, transaction acquisition/release).

Interesting that untyped functional programming and macros allow for a lighter expression of the same basic idea.

Also interesting that maps of functions are a fundamental part of the solution. There are definitions of object-orientation that amount to basically records-of-lambdas. I've long thought that there is a lot to gain by using this basic abstraction as it naturally provides seams for testing/observability/auditing/etc.

Is it just me or is the prismatic blog text being cut off on the side? https://twitter.com/blaines/status/256545302926995456/photo/...

Happens on the iPhone to everyone, I think.

Appears this may be a retina mbp thing...

happens to me on an iPad as well

Very interesting, needs more examples though.

It seems like this fits a similar purpose to Nathan Marz' Storm framework. Could you do a bit of a compare/contrast thing between Graph and Storm?

Graph is not distributed, at least not in the way that Storm is. Notice the post doesn't talk about assigning individual components to individual machines. AFAICT its intended to run on a beefy box, and then horizontally scaled.

You could probably write a higher order function (like "observe-graph" in their example) to distribute the work across multiple machines. Maybe even map it onto a Storm cluster automatically.

Thanks! There are some more complex examples in the slides, I just didn't want to make the blog post any longer.

The main similarity between this and Nathan Marz's Storm framework is that they both rely on a declarative expression of the structure of the computation graph.

However, beyond that there are many key differences. Storm is a graph computation framework, which compiles your specification into a distributed real-time computation pipeline. In contrast, Graph is just a library for expressing composition structure, but says nothing about execution strategy.

In principle (with a lot more code and some more annotations), one could compile Graphs into distributed real-time topologies like Storm. For now we've been using Graph in-process for real-time processing. But because Graph is so simple and close to the language, it's very easy to apply to new situations and build new abstractions on top of. For example, we also find Graph useful for expressing the composition structure of our production services, which are built up from many components.

But, from what I understand, if I structure my software using Graph, it should be much easier to run it using Storm, even without automated tooling. That way Graph becomes a stepping stone: structure your software, and if you need Storm's parallelism/scalibility, define a topology using the same functions.

This is great, but I wish they'd take it one stage further. At the moment, the first example they give (monolithic let) is pretty idiomatic and readable. The transformation from one to the other is pretty automatable. So for most cases, why not just put a macro on top? e.g. defgraph-asfn

Storm requires some manual intervention because you need direct control over how things are distributed.

Nice. Assuming that the computation of each node is memoized, can we 'dirty' a single node, so that during the next computation only the needed parts of the graph are recomputed? This would make multiple computation of models where only a few parameters are changing much more efficient.

Also, has this at all been inspired by SecDB's powerful graph oriented features?

We haven't had the need for that application yet, but it's certainly possible -- that's why declarative specifications are great :)

I haven't heard of SecDB, but I'll definitely check it out -- thanks!

This is definitely a use case I am interested in. I have been playing around with computer algebra in Clojure for awhile where it is critical to keep track of these dependencies (I have a hacky implementation which relies on metadata so that the resulting data structure is out of site). But of course, when you actually want to evaluate something, this is when you need to get sophisticated. There can be multiple paths in the graph to arrive at the same node and you have to decide. With memoization at nodes being an important consideration. Anyway, I would be very happy to see an open source version of this!

P.S. As I understand it all the graphs are dags. In my line of work (which involves doing stuff at the interface of probability and statistics) there are quite a few recursive functions that are difficult to unroll explicitly, or are otherwise undesirable to unroll for the purposes of readability. Can we expect to see circular dependencies at some point. Well, I know, the halting problem and all, but still...

This feels like lazy evaluation bolted on to Clojure (cool!). There's further benefit since you get some dataflow abstraction—which then reminds me of the Par monad.

http://community.haskell.org/~simonmar/par-tutorial.pdf

There's no need to "bolt on" lazy evaluation since (future) and (delay) are parts of the core language (not to mention lazy seqs)

Well, future, delay and lazy sequences are bolt ons to an eagerly evaluated language. I'm not convinced the laziness is Haskell is worth it, tbh, but its true that you really wouldn't need this project.

Usually these kinds of demos are the same as the ones touting the virtues of Haskell's laziness. The visualization is really nice, though, along with the possible paralleization. I think the Par monad is a more interesting comparison, honestly.

http://hackage.haskell.org/packages/archive/monad-par/0.3/do...

Happy to see you're opensourcing. Are you planning to do the same with Flop and Optimize?

Flop is definitely on the list (with about 10 other libraries), but I'm not sure about our optimization code. We plan to start a discussion of what we should prioritize soon, so stay tuned.

Just a heads up, it looks like something is up with your CSS:

http://cl.ly/image/241X1C2D3O35

I'm on 10.8 with the latest version of safari.

In other news, this looks pretty cool.

Are you using a Retina Macbook?

I have the same issue and i'm using a Retina MBP.

Thanks guys. We will look into it.

Same issue on iPad 3 that has retina display

Yup looks like a Retina problem.

I had this same problem on my Android phones Chrome browser and went to my laptop for reading.

If I understand the nature of this system, it's essentially a higher-order function composition tool. Correct?

Nit, defnk? what about def-kw-func or something? not wholly readable as defnk.

Yep, that's correct.

Point taken about the names, thanks for the feedback. Since Clojure already uses defn and fn, defnk and fnk seemed natural. But maybe defn-keyword and fn-keyword would be better.

I really like this idea. Do you have a Common Lisp port of this, out of curiosity? If not I might have to see what I can come up with. :-)

I like to have fully expanded names, ala Common Lisp. My editor will autocomplete those on demand after I use a name once, so it's not a hindrance to reading or writing.

My goal for identifiers is that I can read without consulting my internal lookup table.

Thanks! Nope, no common lisp port -- and we haven't released the Clojure yet, but plan to soon. It's only 150 lines of code or so should be easy to port or rewrite from scratch.

There was a defnk at one time in one of the Clojure libraries (I think it was even created by Rich himself), but it disappeared at some point. Nathan Marz created a defnk macro in Storm that he uses for the same thing. Maybe it's time for Clojure to bring defnk back if it offers some advantages over the :keys destructuring.

Graph reminds me of working with Spring (from what I remember of that). The concepts sound very similar to Spring's auto-wiring.

Reply on news.ycombinator.com