Can anyone share their production experience with Dgraph?
I liked working with it so now I'm the package maintainer for it on AUR [2]. At some point I'd like to make a repo showing how to implement common graph algorithms with the python bindings, since GraphQL+- currently only supports k-shortest path at the query level [3].
[1] https://tinkerpop.apache.org/gremlin.html [2] https://aur.archlinux.org/packages/dgraph-bin/ , https://aur.archlinux.org/packages/dgraph-git/ [3] https://discuss.dgraph.io/t/how-about-doing-some-graph-compu...
I think the latest release is 20.03.1, perhaps time to update?
Will do, I've been a little lax the past few weeks but my school semester just finished so now I should have more time again.
Some constraints we have: * We ingest what some may consider a lot of data - on the order of terabytes a day. This can be 10's of thousands of writes per second.
* We need transactional logic.
* We want to analyze that data as it comes in, so think 10x reads for every write.
GraphDBs out there didn't seem like they would cut it. I eliminated almost every database due to:
* Bad licensing
* Incapable of scaling writes horizontally, or generally anding tons of rights
* No ACID transactions
Most graphdbs out there had at least two of these issues.
DGraph so far has worked really well. We aren't sending it the full load of data yet, so there's still a question around that write load, but at least it's designed for that, and initial numbers have been promising.
The fact that it's liberally licensed, has a really good pricing model, has strong community support, good docs, etc, has made me glad I chose it.
The roughest part is probably the query language, because it's bespoke and therefor ends up having weird unexpected behavior sometimes. Now that it supports GraphQL that should be less of an issue.
But I can comment that getting the "right syntax" was at times extremely frustrating. It has a lot of "there is just one way to do it, and you have to spend a month reading our code to find it" kind of thing going on. It is definitely "beta" software in that regard, and the ease of use of its query language (languages) is abysmal. The documentation is also extremely confusing and incomplete, to say the least. Needs a lot more examples and a lot more "ways to skin a cat" than currently documented.
On a good note, the support provided via discuss.dgraph.io is really good, even though there are so many people struggling to make it do simple things - that support forum will likely be a place where the answer can be found, or someone can answer (rather quickly) with some help.
We were modelling individuals and contacts between them, and the cluster would constantly break with dataset sizes that should have been easily managed. There was clearly something wrong in the storage engine, because we saw insane disk space usage. Dgraph consumed 10s of TB for something that should have taken < 100 GB.
We were one of the largest installations at the time, and were working with the core development team, but they were never able to resolve the issues.
We eventually had to tell management that there was no way we'd be able to operate the thing given its disk space consumption rate, so we had to delay project delivery to rip out Dgraph and replace it with Postgres.
Surely it's better today, but I'll never use it again by choice.
Dgraph is built for performance and with one of our app, we faced similar challenges. After reading some documentation and watching some of their videos, I got to know they compromise space against performance when we have lots of index. We reduced index from 35 to 8 and the db size got drastically low.
I believe you should investigate that as well to check if it's wrong in your database architecture design.
For your gap of <100 GB against 10,000 GB. I assume, you probably created lots of index. Just create good database design and reduce index, you will have low size high performance app.
I’d probably use it for pet projects if they made it easier to automate backups to the cloud, and I’d use it for all projects if they offered a hosted solution.
It's reassuring to see Dgraph undergoing the full Jepsen treatment, even if it highlights that there's still a bit of work to do, and further stability to prove.
It's in our backlog to improve FTS drastically from where it stands today.
So at least for domains where people want to make correlations over data such as a logs, events, transactions, CSVs, etc., I encourage dgraph folks to watch discussions of text closely.
Fun recent example that illustrates this: For ProjectDomino.org (COVID anti-misinfo), we started by ingesting the covid twitter firehose into a graphdb for easy and fast pivoting by tweet/account/etc. However, our analysts need to search by text, and a lot of our current work is now doing ML/graph algorithms to mine the text to infer fuzzy edges: GPU BERT, GPU UMAP, ... . Neo4j supports setting up various text indexes which helps search, but for analytics, we end up having to extract the data out of the DB, infer relationships & scores, and put them back in.