Dragon: A distributed graph query engine | Hacker News Reader

Dragon: A distributed graph query engine

code.facebook.com

150 pointsdineshp224 comments

The query examples looked like Clojure code; is that correct?

The technology of Facebook and Google (where I worked as a contractor) impresses! That said, I like a simpler Internet with smaller scale services like Gnu Social, and simply using email to stay in touch with family and friends. I have this preference both for privacy reasons and also prefer more one on one communication.

It looks a lot like Clojure but I'm not sure it is. For example, this expression:

(filter (> age 20))

That second form to filter should be a function, but in this case it is an expression that returns a boolean and is not itself a function.

>That second form to filter should be a function, but in this case it is an expression that returns a boolean and is not itself a function.

I'm pretty sure this is Clojure or something very close to it. ->> is the thread-last macro. What it does is take the first expression, ($alice) and inserts it as the last expr in the next form so that the second step (assoc $friends) would eval to (assoc $friends $alice). It continues this by inserting this expr into the last part of the next form. So the example would reduce to: (filter (> age 20) (assoc $friends (assoc $friends $alice)))

But as I mentioned, the second argument to Clojure's filter is a function, and (> age 20) is not a function; therefore, this is not normal Clojure (or Clojure or at all). You'd have to replace the standard library's filter function to get the syntax you see in this article, which I doubt they are doing. therefore: not Clojure.

You could also walk the ast and do something special with unknown symbols (and functions that contain them).

i.e. while walking, we notice a function (> age 20) and age isn't visible in scope. We thus rewrite that into something like (fn [age] (> age 20)) or (fn [x] (> (:age x) 20)).

I've taken this approach before with DSLs. It makes for extremely fast to write business logic where business users don't have to care as much about where their data comes from (they just need field names).

In general code, that would be very hard to read, not knowing if a form represented an actual value or a function. Clojure being dynamically typed makes things hard enough; changing the language syntax as you describe sounds pretty risky to me.

The purpose of such a DSL isn't to let you write generic clojure faster. The DSL isn't even targetted at traditional programmers. The purpose is to enable training people who have no idea what Clojure or Lisp to get things done that are complicated (or expensive) to design graphical UX for. These people might have some excel experience or maybe a programming class in college, but are mostly reliant on examples and documentation to get things done. Why teach them things like how to create a function when they don't need to know or care? There is a huge usability difference for these people between (fn [age] (> age 18)) and (> age 18).

Right, it's not Clojure, but we tried to borrow some syntactical elements for readability. More discussion here:

https://groups.google.com/forum/#!topic/clojure/P3eW6Vi2QcU

It has a very similar look and feel to Gremlin (http://tinkerpop.apache.org).

   (->> ($alice) (assoc $friends) (assoc $friends) (filter (> age 20)) (count))

...in Gremlin is:

   g.V(alice).out("friends").out("friends").has("age",gt(20)).count()

It's Lisp code, more particularly Lisp syntax, and I mean that in the most general sense in which Clojure is Lisp syntax. So are Common Lisp, Scheme, Racket and Emacs Lisp.

S-expressions: fully parenthesized prefix notation.

Nice! What are the benefits over TitanDB, Apache Spark/GraphX, GraphLab, Giraph, Neo4j, openCypher Tinkerpop etc?

This one creates optimized indexes based on the social graph's intrinsic semantics.

Seems like the big improvement is heuristically keeping related data on the same hosts as much as possible, to improve query performance.

This sounds really cool.

For distributed relational or graph databases, seems like the key trick for making queries efficient is to get related data on the same host, whenever possible.

So would be cool to dig into the specifics of the algorithms they are using, to see exactly how they are optimizing where to store the data. With a graph database, it's impossible to guarantee having all related data together (eventually, friend of a friend of a friend...will be on a different host). So needs to be heuristics based.

For tree shaped data, on the other hand, it is possible to have the root of each tree and all of its related data on the same host (assuming each tree is "reasonable" size). Google's F1 project took this approach.

Hello, I wrote the blog post. There is a link to a video in the blog and we published something at NSDI last week:

https://www.usenix.org/conference/nsdi16/technical-sessions/...

Thanks for the link! Looks like an interesting read.

looks interesting, where can I download Dragon? :D

I don't think it's open source yet. Maybe in the near future

Slight tangent here but is it me or has the experience in social networks taken a hit? Feels like it was better with low complexity hack technology than with technology solutions focused on scaling the experience. I sometimes wonder if we are changing experience too much for technology optimization.

- LinkedIn has been terrible for about 2-3 years since things stopped updating real time and the timeline started getting random

- Facebook randomly suggesting things from your graph based on what it thinks you like

- Twitter I heard soon will be no longer in chronological order?!?

I think there is something to be said for low complexity MySQL and memcached.

I don't think any of those things are caused by optimisations or needing to scale.

Saying that Facebook is recommending things simultaneously at random, based on a reccomendation algorithm, and also as an optimisation which wouldn't be needed if they used MySQL, is obviously nonsense.

@chrisseaton - big fan of your work, eagerly awaiting Truffle JRuby production ready release.

I'm not saying MySQL > some specialized solution for scale... I'm guess I'm drawing the correlation that when companies have to make investments like this they are also trading off experience, or maybe to your point screwing up the experience using recommenders and algorithms instead of letting my feed flow.

Yeah I see your clarification in your other comment where you say that when they start to scale it opens the door for tons of other complexity. That makes more sense.

I don't think that's to do with stack complexity, more like:

"Crap how do we make money"

"Crap how do we retain users/keep them here longer"

The stack is to do with providing a fast performance to vast numbers + making it easier to work on for engineers than it might be at that complexity.

yeah I hear that and to be clear I'm not blaming all their experience issues with the stack.

But there are definitely stack complexity trade offs when you get to the scale of these guys.

Also could be more correlation not causation that when you get to a certain scale and more stack complexity you also start hiring specialized staff that end up diluting the culture and spirit of a startup.

Reply on news.ycombinator.com