Amazon Neptune – Fast, reliable graph database built for the cloud
aws.amazon.com
aws.amazon.com
If you're in town and into this stuff, ping me at leo [at] graphistry, and would love to catch up Th/F for coffee+drinks. Also here + email, of course!
Amazon owns the BLAZEGRAPH trademark: https://www.trademarkia.com/blazegraph-86498414.html
Blazegraph's CEO is currently at Amazon as Principal Product Manager: https://www.linkedin.com/in/bradley-bebee-a15764b/
Then will Wikidata have to migrate to AWS to maintain the endpoint?
And then after two years, when you're no longer startup with 100usd bill, but bigger company, you're completly tied to a jungle of amazon products, and your exit strategy is very very costly.
clever amazon, clever.
Consider the evolution of container hosting services: first we had PaaSes like Heroku with proprietary container formats; then we got Docker, but Docker Swarm was nascent and there was no serious Docker Swarm IaaS-cloud offering. But then, very quickly, AWS built ECS; Google responded with Kubernetes; and then Kubernetes became the open standard, made everyone forget about Docker Swarm, and took over (and is even replacing ECS now.)
That's what happens when AWS enters a space. And it's great.
No lock in at all.
If you factor in the cost of not taking "the easy way" you'd likely never get past the 100usd phase.
You're point is valid. I'm just suggesting the lens / context isn't as one-sided as you've presented.
Put another way, plenty of startups and VCs would love to have "getting out from under AWS" at the top of their good problems to have list.
This tool proposes to design query patterns from a graph data model, via drag n drops. The tool can then compile the patterns as SPARQL, run them on an endpoint and format the results as map/forms/tables/graphs/HTML (via templating)/...
Another service of Datao (http://search.datao.net) proposes a search-engine view of those queries so you can type the textual representation of an object in any public SPARQL endpoint, and the service will list the queries currently available in Datao that can be applied upon this object. You can then run these queries with a click, and get the HTML templating of the query results.
Feel free to have a look at the website, if you find any interest in this tool. ANy feedback is welcome.
PS: Sorry for the poor quality of the videos. I manage this project on my spare time :)
What do you mean by "unfair" here?
Support for various storage backends:
- Apache Cassandra®
- Apache HBase®
- Google Cloud Bigtable
- Oracle BerkeleyDB
I don't understand how a database doesn't have its own native store. What exactly does a graph database actually do if it doesn't manage the data fed to it? Same is true for CayleyGraph† https://github.com/cayleygraph/cayley and proabably others.†Plays well with multiple backend stores:
- KVs: Bolt, LevelDB
- NoSQL: MongoDB
- SQL: PostgreSQL, CockroachDB, MySQL
- In-memory, ephemeral1) "native" graph db Neo4J is an example of this. This takes advantage of index-free adjacency. Each node knows what other nodes it is connected to and hence traversals are very fast. The issues you run into are when you try to scale. Data that fits onto a single machine is fine and you can replicate your data for fast parallel reads/traversals across disparate regions of a massive graph. However you no longer have the concept of data sharding and distributing the graph as index-free adjacencies don't translate across physical machines. And another drawback is highly connected vertices, you will expend a tremendous amount of resources deleting or mutating a vertex with, say, 10^6 edges. But that vertex is probably a bot so you should delete him anyway.
2) inverted index graphs, non-native graphs, whatever anti-marketing name it might have. These rely on tables of vertices and other tables of edges. Indexes make them fast, not as fast for reads but very fast for writes. And you get distributed databases (Cassandra, for example, a powerful workhorse of a backend with data sharding and replication factor, etc.). But then you have to yet another index to maintain and the overhead can get expensive. This is the model adopted by DataStax, who bought Titan DB (hence the public fork to Janus) and integrated it with some optimisations and enterprise tools (monitoring etc, solr search engine) to sit on top of Cassandra.
Both now have improved integration with things like Spark. Cypher is probably faster than Tinkerpop Gremlin especially with the bolt serialisation introduced in recent versions of neo4j.
So janus is the graph abstraction layer of the second type and so needs somewhere to save these relationships. It all comes down to use case (and marketing) to decide what works best for you.
* https://www.datastax.com/dev/blog/a-letter-regarding-native-... (tldr; there is no such thing as a native graph database)
* https://neo4j.com/blog/note-native-graph-databases/ (tldr; native graph databases do exist)
Regarding Cypher vs Gremlin: serialization could be a thing but what matters among other things are efficient query optimizations, algorithm and (physical) data model. Ultimately, databases are all reading from 1-dimensional spaces (RAM or disk), either randomly or (best) sequentially. If you can colocate vertices with their respective edges, you're fine: this is trivial for graphs with no edges or graphs that form a linear chain. If not, then things start to become fun, especially in a distributed way. This will impact performance; the language, not so much.
Graph database projects are often times just an adapter for doing Graph queries on-top of another store.
At their core, a graph database can be reflected simply with just documents and adjacency lists https://en.wikipedia.org/wiki/Adjacency_list
It provides tools to run complex queries on graphs, and manages data models and indices to execute them fast.
However, when we get to manage the storage layer (Bolt, Level -- that's being generalized into local-KVs in the next release) we get to build our own indexes for better data management and performance. But there's no reason we can't hand that job off either -- hence supporting multiple (remote) backends. For the local stuff, though, at some point, Bolt is just a very good BTree implementation.
Even propagation and node invalidation are awesome for rapid what-if style experimentation, and I am so psyched to see more and more attention being paid to graph computing in general.
[1] https://www.github.com/kchoudhu/openarc [2] https://www.anserinae.net/whats-cooking-openarc-edition.html...
Is it fair to say that traditional RDBMS/SQL are for storing different "sets" of related information (tables for products, users, orders).
Graph databases are for storing data about the _same_ set of data as it interrelates to itself.
- a User and and all their Friends (who are also users) - a Keyword and all associated Terms (which are also keywords)
Is that right?
Just as RDBMS can have tables about different things (Products, Users, Orders), graph databases can use labels on nodes for different things (so you can have :Product nodes, :User nodes, :Order nodes). Though with graph databases, there is often less rigidity in the associated data than in RDBMS, as there is no requirement for explicit schema for properties on nodes of different types in a graph db (plus you can multi-label nodes).
The real differentiator is how relationships are modeled, and how they're traversed in queries.
With RDMBS/SQL you're going to be working with data in tables, and use join tables as the relationships between them. You're likely going to need to be explicit about what is being joined together, so the relationship chain is likely to be very rigid.
With graph databases, relationships and relationship traversal is used in place of join tables and table joins, which gives much more flexibility over how to traverse. You can certainly do friend-of-friend-of-friend queries much more easily, but you can also perform variable-length traversals using custom logic for which nodes are in the path and which relationships are traversed (type, direction, and count), and that can be very well-defined, or very loosely defined, or a mix, as needed. I don't believe there are good ways to do that kind of ad-hoc table joining in RDBMS.
As an example of very loosely defined traversals in queries, you can ask for a shortest path between two nodes, knowing nothing about the nodes or relationships that could be between them, and get a path back showing the connecting nodes, with the relationships between the nodes providing context.
Has anyone ever considered the licensing implications of this? How is amazon able to convert an open source product into a proprietary one and then charge for access to it?
Of course you can argue they’re charging for the infrastructure management, not the software itself. But that argument quickly breaks down as Amazon introduces new software, under new names, with a proprietary management interface over an open source core. Try to find the source code; you can’t.
And if you accept the premise that they’re just charging for hosting, then it leads to the question of why an open source project doesn’t reap any benefits from that hosting, or at the very least, from the management interface on top of it.
It seems like a better solution would be something akin to AWS marketplace, where open source projects are available to be hosted, and the maintainers can see some revenue from them.
It seems like unfair rent seeking behavior that amazon is able to slap a management interface on open source software and then charge for it under the guise of “hosting.”
Totally no problem with liberal licensed open source software.
This is also the intended behaviour of such licenses.
Also many of those big bad commercial companies contribute back big time to a number of projects. Why? I guess sometimes because devs want to and also because it makes sense business wise so they don’t have to maintain the code themselves.
Seems like a management interface is a clear cut derivative product. Where’s the source code?
Or perhaps amazon does consider licensing and only builds on top of, eg Apache licensed projects?
The newer AGPL closes this loophole.
And yes: except for Linux and the GNU tools I guess most companies stick with Apache, BSD, Eclipse and MIT licensed software.
the question in amazons case is that since they sell the infrastructure, they can and probably do undercut any competing providers by charging themselves less.
so, it seems monopolistic. otoh their service is good and their customers get at least reasonable prices. so ... ?
http://radar.oreilly.com/2007/07/the-gpl-and-software-as-a-s...
Amazon / Google / etc are not redistributing the software as it is running on their servers in their environment, therefor, there is nothing wrong with the existing licenses.
Are they making money with software they didn't build? Yes, but so are we.
To be clear, it’s not the hosting of open source applications I see as the problem, but the closed source management/orchestration software built on top of it.
a) Slower than RDBMS/NoSQL but still pretty respectable, so it's a good choice for things like offline analysis.
b) About the same at RDBMS/NoSQL, so you could use it to handle production traffic if you want.
c) Faster, so you should definitely prefer it in production, e.g. for fetching upvotes and comments on posts.
For efficient graph DBs it's better to have a lot of ram and cores ...
Btw, anyone knows how such solutions handle cross machine traversals? Are they schema-based? So the DB knows how to manage data locality and efficient joins/traversals?
Worth to take a look if you need a managed Gremlin solution with some degree of global distribution.
I really hope Amazon will propose a facility to retrieve the RDFS data model of an endpoint in a uniform way.
How would I bolt on an inference engine to this if none is offered, i.e. to provide OWL:RL?
My point is that gremlin has been super efficient for us to express (in its functional way) tricky traversals. So I do not see any reason to discard it as a "inefficient" technology.
(I didn't downvote you though, it's a common misconception.)
SPARQL is supported.
If you want to try it out:
pip install graphql-compiler