Jepsen: Dgraph 1.1.1
jepsen.io
jepsen.io
I'd like to thank Kyle in doing another round of testing. Some of these bugs that we fixed (in 2018 and 2019) were very tricky edge cases -- it's incredible to see Dgraph running so much more stable now, under all varied failure scenarios.
Let me know if you have any questions, I'm around to answer.
Would love some insight into the plans or use case for the plain GraphQL endpoint.
Dgraph can also be called via Apollo Gateway, as part of their federation.
We also have other very interesting stuff in the pipeline, which would cater a lot to front end development as well as backend. So, stay tuned.
We use 100ms heartbeats, with 20x that for election timeouts, in line with Etcd. Don't think that's exposed to the end user -- nobody has needed to tweak that.
https://discuss.dgraph.io/t/hiring-community-engineer-to-sta...
(Thank you for sharing your great work.)
Also, are you an acquaintance of Kyle Kingsbury? Just curious as you refer to him by his first name.
I've wanted a robust, easy-to-use graph database for years (ever since reading about the crazy brilliant graph database stuff that goes on inside Facebook https://www.facebook.com/notes/facebook-engineering/tao-the-... ) and Neo4J never really cut it for me.
Watching Dgraph mature - and survive two rounds of Jepsen with relatively decent marks - is intriguing. I don't have a project that needs it yet but maybe something will come up soon.
N4J: "We sell cars. Our cars can be anything you want, and are at least pretty good at everything, and often great and even better than any other possible car you can get!"
Client: "Cool, I need a spacious four-door sedan with good gas mileage that's also a race car"
N4J: "Oh yeah, definitely, no problem, this is the product for you, totally fits your use case!"
Client orders car, it arrives, is in fact a delivery van that likes to stall at traffic lights
Client: "Uh. This would be useful if I needed a delivery van for a route with no stops, I suppose."
What did you find lacking in Neo4j?
Turns out the official Python client library is five years old now so I clearly need to update my mental model of what it can do!
I thought Dgraph was going to be the "secret sauce" for my app after reading the list of features (maybe I was mesmerized by the cute mascot). Few months down the line, though, sometimes I question my decision and whether I should have used the good old PgSQL. Let me explain.
1. Coming from SQL and key-value NoSQL, Dgraph was very foreign to me. It's like OOP guy learning Functional Programming. Now I'm quite comfortable with it, but it took me months to be productive. To give an analogy, learning Dgraph is like learning Scala instead of Python.
2. Actually it's worse, the resources are mostly in the forum (micheldiz is my hero) and the official website. Until recently, the navigation for the docs is horrible, everything in one big HTML file, difficult to jump in from Google search result.
3. Rudimentary tooling for development phase. When you're working on a new idea for a product, you will experiment A LOT with the schema and data. When you write wrong data, with most RDMBS you can use GUI to right-click and delete or edit. In Dgraph, you must write mutation query (assuming you remember the syntax, as it is a bespoke language). Dgraph GUI is very minimal.
4. About the mutation.. in Dgraph (as far as I know) there's no referential integrity in the DB-sense. Like, you can make FK to non-existent object, or insert something invalid without returning error (but it's not stored, since it's invalid). The "integrity check" is in your app.
5. Because of #3 and #4, I find it easier to just drop the whole database and recreate it along with seed data, every. time.
6. The documentation for the Java client library is very minimal. So there you go, unfamiliarity with the query language ("GraphQL+-"), with the Dgraph itself, and with the client library.
I still use Dgraph, it's a good fit for my app, but if you're starting on a new business idea, maybe don't use anything fancy. My mistake, being a developer, I mixed research (new tech!) and bootstrapping.
(In case you're wondering, my app is http://s.id/axtiva-android-test -- still version 0.0.x, but recently I'm releasing weekly)
Though, I can see part of your pain is because of the custom query language, GraphQL+-. We now offer standard official GraphQL compliance as well, which tackles a lot of these issues you ran into.
1 and 2. GraphQL is becoming very common, so plenty of resources.
3. GraphQL has many amazing editors.
4. GraphQL allows for lack of referential integrity by setting certain fields in the object as non-nullable, which can remove those objects from the results and so on -- which is frankly, the thesis we have around a distributed graph database, sharded by predicates (not nodes).
5. arrr... sad.
6. GraphQL has many client libraries and so on.
Hope we could change your opinion by switching you to standard GraphQL -- particularly, if you don't need the advanced features provided by plus-minus.
Thinking of the graph database as a graphics programmer might think of the GPU might be a helpful perspective.
[1] https://cayley.io/ [2] https://oss.redislabs.com/redisgraph/
EDIT: I love dgraph, and reevaluate it anew frequently. However, I've shipped cayley and this approach works. Continue down-voting this throwaway account though.
One was for event saving and retrieval, which was able to sustain a stable 60k writes /s with simultaneous 10k reads /s. It worked great overall, with stuff needing nontrivial tuning being 1. RAM usage 2. If you overwhelm it with writes it'll stall to keep up with level 0/level 1 compactions.
Another one is OctoSQL[1], where we're building exactly-once event-time based stream processing all around Badger. So far it was a breeze and I don't think we'd build it if not for Badger.
Overall, at least the storage engine they're using is awesome, and I can definitely recommend it!
I liked working with it so now I'm the package maintainer for it on AUR [2]. At some point I'd like to make a repo showing how to implement common graph algorithms with the python bindings, since GraphQL+- currently only supports k-shortest path at the query level [3].
[1] https://tinkerpop.apache.org/gremlin.html [2] https://aur.archlinux.org/packages/dgraph-bin/ , https://aur.archlinux.org/packages/dgraph-git/ [3] https://discuss.dgraph.io/t/how-about-doing-some-graph-compu...
I think the latest release is 20.03.1, perhaps time to update?
Will do, I've been a little lax the past few weeks but my school semester just finished so now I should have more time again.
Some constraints we have: * We ingest what some may consider a lot of data - on the order of terabytes a day. This can be 10's of thousands of writes per second.
* We need transactional logic.
* We want to analyze that data as it comes in, so think 10x reads for every write.
GraphDBs out there didn't seem like they would cut it. I eliminated almost every database due to:
* Bad licensing
* Incapable of scaling writes horizontally, or generally anding tons of rights
* No ACID transactions
Most graphdbs out there had at least two of these issues.
DGraph so far has worked really well. We aren't sending it the full load of data yet, so there's still a question around that write load, but at least it's designed for that, and initial numbers have been promising.
The fact that it's liberally licensed, has a really good pricing model, has strong community support, good docs, etc, has made me glad I chose it.
The roughest part is probably the query language, because it's bespoke and therefor ends up having weird unexpected behavior sometimes. Now that it supports GraphQL that should be less of an issue.
But I can comment that getting the "right syntax" was at times extremely frustrating. It has a lot of "there is just one way to do it, and you have to spend a month reading our code to find it" kind of thing going on. It is definitely "beta" software in that regard, and the ease of use of its query language (languages) is abysmal. The documentation is also extremely confusing and incomplete, to say the least. Needs a lot more examples and a lot more "ways to skin a cat" than currently documented.
On a good note, the support provided via discuss.dgraph.io is really good, even though there are so many people struggling to make it do simple things - that support forum will likely be a place where the answer can be found, or someone can answer (rather quickly) with some help.
We were modelling individuals and contacts between them, and the cluster would constantly break with dataset sizes that should have been easily managed. There was clearly something wrong in the storage engine, because we saw insane disk space usage. Dgraph consumed 10s of TB for something that should have taken < 100 GB.
We were one of the largest installations at the time, and were working with the core development team, but they were never able to resolve the issues.
We eventually had to tell management that there was no way we'd be able to operate the thing given its disk space consumption rate, so we had to delay project delivery to rip out Dgraph and replace it with Postgres.
Surely it's better today, but I'll never use it again by choice.
Dgraph is built for performance and with one of our app, we faced similar challenges. After reading some documentation and watching some of their videos, I got to know they compromise space against performance when we have lots of index. We reduced index from 35 to 8 and the db size got drastically low.
I believe you should investigate that as well to check if it's wrong in your database architecture design.
For your gap of <100 GB against 10,000 GB. I assume, you probably created lots of index. Just create good database design and reduce index, you will have low size high performance app.
I’d probably use it for pet projects if they made it easier to automate backups to the cloud, and I’d use it for all projects if they offered a hosted solution.
So at least for domains where people want to make correlations over data such as a logs, events, transactions, CSVs, etc., I encourage dgraph folks to watch discussions of text closely.
Fun recent example that illustrates this: For ProjectDomino.org (COVID anti-misinfo), we started by ingesting the covid twitter firehose into a graphdb for easy and fast pivoting by tweet/account/etc. However, our analysts need to search by text, and a lot of our current work is now doing ML/graph algorithms to mine the text to infer fuzzy edges: GPU BERT, GPU UMAP, ... . Neo4j supports setting up various text indexes which helps search, but for analytics, we end up having to extract the data out of the DB, infer relationships & scores, and put them back in.
It's in our backlog to improve FTS drastically from where it stands today.
It's reassuring to see Dgraph undergoing the full Jepsen treatment, even if it highlights that there's still a bit of work to do, and further stability to prove.
that means that if you have externally unique IDs that you have infrastructure around, you are either caching that node's UID externally or doing an XID->UID lookup in order to create edges.
there is a bulk loader but that's only available in HA mode, and the UID:XID map it generates is obviously for data you already had in flat files (or whatever). so it's ok for static data sets, but not ideal for live updating data.
the gRPC API also has strange undocumented (AFAICT) behavior where even smallish batches of 100 hit some unspecified gRPC limit, so you need smaller batches ergo more commits ergo more wasted compute.
There is. You can lease UIDs from Zero, and do your own assignment. Look at /assign endpoint [1]
> doing an XID->UID lookup in order to create edges.
Also, you can use upserts to do an XID lookup, before creating a new node. Which is practically what other DBs do too.
> there is a bulk loader but that's only available in HA mode
Don't know what that means. Bulk Loader is a single process (not distributed), and can be used to bootstrap a Dgraph cluster. The cluster can be a 2-node cluster, or an HA cluster, that doesn't matter.
> where even smallish batches of 100 hit some unspecified gRPC limit
Never heard of that. Grpc does have a 4GB per message limit. But, I doubt you'd hit that with 100 records.
i hope this comes across as non-critical feedback, but it'd be really, really nice to put that assign endpoint in some form or fashion in the Mutation documentation. it is completely absent from there, and i don't recall seeing it in the tour of dgraph either.
furthermore, it's absent from the golang client. the documentation states:
> It’s possible to interface with Dgraph directly via gRPC or HTTP. However, if a client library exists for you language, this will be an easier option.
however it looks like i'll need an additional HTTP layer to interface with the /assign endpoint. not a huge deal, but that seems like a big functionality gap with the golang endpoint - would definitely like to see that added in there.
lastly, the /assign endpoint and the bulk loader can only be run with a DGraph Zero instance, which, as far as i can tell, which doesn't run by default with the provided docker image. that's an important detail that's not super duper obvious from the docs, until you start seeing parameters like dgraph-zero, and then realizing that it doesn't come with the quick start docker image.
again, hope this isn't taken personally. thanks for your work on the project!
Assign endpoint is something that you can just do once. You could say, give me a million UIDs, and then use them however you want. You don't need to call it repeatedly.
Also, its an endpoint to Zero, not to Alpha. Zeros are not supposed to be directly talked to, in a running cluster. We're now doing work around exposing some of Zero endpoints via Alphas, in our GraphQL rewrite of the /admin endpoint. So, that might make it easier.
I think the consistent theme I'm hearing here is that our documentation isn't clear -- we aim to improve that. But, could use more critical, logical feedback / suggestions on our forum -- so please feel free to pitch in there.
I know almost nothing about graph databases, so presumably this is just my ignorance. But if entity-focused retrieval is an important use case, isn't clustering by attribute going to kill performance? Naively, I'd think that one would cluster by entity and what an entity is connected to.
Also keep in mind that typical Dgraph workloads request specific attributes (think "SELECT NAME, AGE") rather than everything ("SELECT *"), which reduces the impact of fanout. :)
Those can be done concurrently if at the same query level, so not necessarily any slower. In other terms, the number of network calls required (in a sufficiently distributed cluster, where each predicate/attribute is on a different server), is proportional to the number of attributes asked for in the query, not the number of results (at any step in graph traversal).
And that's the big part of the design. By constraining the number of network calls to very few machines, while doing traversals, which would lead to millions of results in the intermediate steps -- Dgraph can deal with high fan-out queries (with lots of node results) much better.
Alternative would be to shard by nodes (entities) -- in which case, if the intermediate steps have millions of results, they could end up broadcasting to the entire cluster to execute a single query. That'd kill latency.
So, the problem is not how many attributes a query is asking for -- that's generally bounded. The problem is how many nodes you end up with as you traverse the graph, those could be in millions.
That's why many graph layer systems suck at doing anything deeper than 1 or 2 level traversals / joins.
> Those can be done concurrently if at the same query level, so not necessarily any slower.
An important clarification, yes! I should have made that more explicit. :)
Imagine, in an abstract way, that it is a hashing process. Where the attributes are spread over several instances in the cluster and Dgraph just "decodes" it for you quickly because it already has the key.
Could it be based on entities? yes, but I don't think it would bring any benefit. Imagine that you have 10 instances, and you have 10 entities. You have 3 entities with many edges (values and outgoing edges) and 8 "light" entities, without much information. Well, you will leave 3 entities in high usesage of resources while 8 entities are idle because they do not have much information to deliver to all query and mutation requests in the Cluster.
A system based on attributes in a distributed way is much more performance tho.
> Naively, I'd think that one would cluster by entity and what an entity is connected to
In a distributed system, that approach leads high-fanout and network broadcasts, which kills query latency. Ideally, you want to do a traversal / join in one network call (max), not more. Because of this design, Dgraph can execute arbitrary depth queries in a much faster way. More details are in [1].
https://github.com/jepsen-io https://jepsen.io/ https://jepsen.io/ethics
For simulation testing, I'd suggest looking at Maelstrom, which uses Jepsen to provide a sort of workbench for writing toy Raft implementations in any language. You give it a binary which takes messages as JSON on STDIN and emits messages to STDOUT; it spawns a bunch of "nodes" (local processes) of that binary, connects them via a simulated network, generates pathological network behavior, simulates client requests, and verifies the resulting histories with Jepsen.