Graph databases and Python
de.slideshare.net
de.slideshare.net
I've used graphdbs in the past but a nice collection of patterns and best practices would be nice - upping my game on this topic is a current interest of mine!
This is a good book for a quick intro to graph dbs for anyone interested.
Also, all the code you see on the blog and then some is open sourced and available on github => https://github.com/maxdemarzi?tab=repositories
* TinkerPop Book "Resources" section: http://www.tinkerpopbook.com/
* Marko's blog: http://markorodriguez.com/ -- start with the "On Graph Computing" post (http://markorodriguez.com/2013/01/09/on-graph-computing/).
* Aurelius Blog: http://thinkaurelius.com/blog/
For Python, see the Bulbs Docs: http://bulbflow.com/docs/
I've been meaning to update the Bulbs docs for Bulbs/Titan. It's essentially the same as Bulbs/Neo4jServer and Bulbs/Rexster, except Titan does indexing a bit different.
Here's a few pointers...
* Boutique Graph Data with Titan: http://thinkaurelius.com/2013/11/24/boutique-graph-data-with...
* Titan Overview: https://github.com/thinkaurelius/titan/wiki
* Download: https://github.com/thinkaurelius/titan/wiki/Downloads
* Titan Server: https://github.com/thinkaurelius/titan/wiki/Rexster-Graph-Se...
* Bulbs Titan Example: https://gist.github.com/espeed/3938820
2) make sure you have the API in mind
3) choose a problem
4) take a pen
5) take sheet of paper
6) solve the probem
7) ???
8) profit!
There is most of the time (to not say always) several solution to solve a problem, but only one particular solution will be the best...
The thing is there is still no FOSS projects in the wild using graphdbs from which you can copy the designs, but is it really what you want?
Also, on the neo4j.org page, the claim that "graph data model['s] expressiveness supersedes the relational model" seems a little bit spurious, seeing as, as I understand it, the relational model and graph data are both anchored in first-order predicate logic, and therefore should be able to do the same things essentially (although Codd-style RDBMS with a little bit more fuss regarding the necessary schemas).
One of the leading native graph processing engines is GraphLab (http://graphlab.org/); however, the creator of GraphLab, Dr. Joey Gonzalez, is now focused on GraphX, which is essentially GraphLab built on Spark (http://spark.incubator.apache.org), which is a non-native analytics platform.
Building a graph-processing engine on a general processing system like Spark makes pre-processing and post-processing much easier.
See "Introduction to GraphX - Presented by Joseph Gonzalez, Reynold Xin - UC Berkeley AmpLab 2013" (http://www.youtube.com/watch?v=mKEn9C5bRck)
Also, a bunch of advancements in graph processing are coming down the pipe, which will be released in a few months (see https://news.ycombinator.com/item?id=6786563).
Ditto for "native" versus "non-native" graph storage.
See this post by Dr. Matthias Broecheler, the creator of Titan (https://github.com/thinkaurelius/titan/wiki)...
"A Letter Regarding Native Graph Databases" (http://thinkaurelius.com/2013/11/01/a-letter-regarding-nativ...)
Furthermore, please have a look at "On Graph Computing" for a break down of 3 different categories of graph computing systems -- toolkit, database, analytics. http://markorodriguez.com/2013/01/09/on-graph-computing/
Finally, yes -- there is no theoretical expressivity gains between RDBMS and property graphs (and, RDF graphs). Nor is SQL (Turing Complete versions) any less expressive than Gremlin (Turing Complete path recognition). The only argument you can make is that graphs are more (or less) effective in terms of conciseness of expression and speed of execution at particular problems. Typically (as expected), its the difference between problem datasets that look like networks (graphs) and those that look like spreadsheets (tables).
Modelling in graphs is new to me so I was wondering if anyone had any tips or pointers.
Bulbs Python Client: https://github.com/espeed/bulbs
Then I made a rails front end using the acts-as-sane-tree gem, which is designed to use this postgres data model and recursive queries: https://github.com/chrisroberts/acts_as_sane_tree
It's quite fast.
You should be able to say:
CREATE EXTENSION ltree;
You can use Gremlin with any TinkerPop/Blueprints (https://github.com/tinkerpop/blueprints/wiki) enabled graph database (which means almost all graph DBs).
Good point. Relational databases have been used for BOM (Bill-Of-Material) modelling (a manufacturing application) for ages. DB records representing a manufactured product or component can have fields that point to other record(s) in the same table, which can be child components of the product. E.g. airplane -> engine, wings. Engine -> engine parts. Wings -> wing parts. Etc. And this can be recursive.
Another such example is when you want to model an employee entity, where a manager (who has employees - or reports) is also an employee.
http://stackoverflow.com/questions/16759606/is-there-a-equiv...
Here's how to use server-side Gremlin scripts in Python with Rexster, which is TinkerPop's open-source server that runs multiple graph databases, including Neo4j...
https://groups.google.com/d/topic/gremlin-users/Up3JQUwrq-A/...
Tinkerpop people are pushing too hard Gremlin DSL/API/whatever which is AFAIK only useful in some situation somewhat complex and more or less a nice way to write some common queries. But in simple situations any language with the raw Graph API can do the job. And there is still no drivers for Python in Rexster. I tried, but it was too complicated. Rexster itself is too complicated.
Neo4J with their own query language made things even more complicated. Instead of a “Graph that can be queried with your preferred language” you get a “Graph that can be queried with something that looks like SQL but is not“
ArangoDB is nice for people that want to do JavaScript full stack. Which is not the case of people doing Python.
Also, there is nobody marketing graphdbs just saying “it solve the general problem“. period.
The only thing that may hold you back from using graphdbs are performances but in a lot of situtations you don't care especially in situations where you want to be flexible and to move fast. That's where GraphDBs shine a lot. Of course there is also the graph/tree problem solving space but this is taken for granted.
GraphDB actors market a lot the specialized database aspect of graphdbs, nonetheless graphdbs are good even for solving generic webdev problems.
Also if you are looking for a Graph Database server that does just that, and where you can query the graph in Python 2.7 (or Scheme) have a look at: https://github.com/python-graph-lovestory/Java-GraphitiDB
http://jugad2.blogspot.com/2013/01/graph-tool-python-module-...
http://blog.neo4j.org/2013/10/the-first-graphgist-challenge-...
Then maybe you can save all links from a node in the node, so you can get all the links with one read access. Fine. But as soon as you get to the second or third level, I would expect the magic to be gone. Say every node has 100 links. OK, so the first 100 links you get in constant time c. But to get the second level, you already need 100 requests (one for each node and it's attached link list). So 100c time. For the third level you need 10000 reads, 10000c time. The next level would be 1000000 requests.
Just saying I'd expect things to get ugly with a graph database pretty fast, too (not as fast as with a relational db, but still).
I haven't really coded a big graph based app, but my expectation would be that get really good performance, a hand coded solution would always be required. For example trying to squeeze as much of the relevant data into memory in a compressed way. Am I wrong?
Oh and also I am not sure how good relational DBs are at query optimization. Just because the visible model is "one row per link" doesn't mean the db couldn't do some intelligent caching internally.