neunhoef

Yes, it has.

In academia the cost for running computers and software deployments is usually greatly underestimated (I speak from experience). If you are a for-profit company and compute the full economic costs of running a database deployment including training, devops, alerting, 24/7 maintenance, upgrades, backups and everything you easily come to some $60000 per year.

Managed services can beat this price by a huge margin, although they seem expensive at first glance.

(ArangoDB developer here) MongoDB is a document store, ArangoDB is multi-model, so you have graphs and key/value and search as a additional benefits. The query language AQL is also a huge plus. Price comparisons are always hard, but for similar deployments Oasis will be a bit cheaper. One really has to look at the details here, in particular with respect to sharding and resilience.

Disclaimer: I am one of the core developers of ArangoDB and I work for the company ArangoDB.

Yes, ArangoDB is young (6 years) in comparison to PostgreSQL (30 years).

Yes, PostgreSQL is a phantastic database with an amazing open source community and is not going away any time soon.

Yes, PostgreSQL is a good choice for a project in which you need a relational single server database.

However, ArangoDB actually has a different value proposition, so in a sense, it is not a direct competitor.

ArangoDB is native multi-model, which means it is a document store (JSON), a graph database and a key/value store, all in one engine and with a uniform query language which supports all three data models and allows to mix them, even in a single query.

Furthermore, ArangoDB is designed as a fault-tolerant distributed and scalable system.

In addition it is extensible with user defined JavaScript code running in a sandbox in the database server.

Finally, we do our best to make this distributed system devops friendly with good tooling and k8s integration.

Last but not least, ArangoDB is backed by a company which offers professional support.

Therefore any well informed decision for a project needs to look at the value propositions and capabilities and not only at the age and experience, which is of course a big argument, since people are - rightfully so - conservative with their databases.

As explained in the article: The "library part" of our executables is very small in comparison to the executable size. Furthermore, the memory usage of the database itself is usually much greater than the size of the executable itself. Finally, one rarely deploys multiple instances of an ArangoDB server on the same machine, so savings by shared libraries are also not that great.

Therefore the "waste" is very minor and the advantages outweigh the slight increase in memory usage.

After compilation the executables have symbols and debug information. For the release we simply strip it. It is easy to get/provide versions with debugging information. We plan to provide deb and rpm packages containing these and provide separate debug packages. This works but is not yet published

Author of posted article here: thanks for the additional pointers. It seems that graphistry excels at visualization. Essentially, your offering confirms the main story of the article: make more out of your (graph) data by extracting it from Hadoop to a different tool.

And obviously, one should use the right tool for the purpose . I think graphistry is a good choice for graph visualization, graph databases like ArangoDB or Neo4j will be good at ad hoc traversals. And multi-model databases like ArangoDB or OrientDB will be good at a wide range of ad hoc queries. Anyway, thanks again for the pointers.

Does anybody reading this have any experience with the performance of AWS Neptune? I would be very interested to hear about the performance of deep graph traversals on large sharded graphs.

NoSQL is a very wide field. There are lots of special cases in which certain NoSQL data stores can help a lot. One of the points the posted article makes is that native multi-model databases help you not to sacrifice relational and yet reap some of the benefits certain NoSQL data models (like graph) can provide.

In discrete mathematics this happens a lot. Group orders, sizee of conjugacy classes, semigroup orders, numbers of isomorphism classes, character degrees, etc.

You are of course right. My mistake.

Silly solution: The rationals are countable. Just use any bijective mapping of the natural numbers to the rationals and then use the scheme for the natural numbers. (I know, a mathematician's answer! :-) )

That is right, if you have access to your comparator, then this is much easier. However, it would still not work well for negative numbers, and, in many cases you sit client side and do not have access to your database engine.

Great. Thanks. I was not aware of this.

Many JSON parsers will treat long numbers as doubles. Therefore the only safe way to store them without potential loss is in a string. Without the encoding in the article or something along these lines, the sorting done by a database is not compatible with the integer sorting.

Author here: In mathematics, in particular discrete mathematics, there is a genuine need to store long integers accurately and to compare them numerically. This encoding is useful to do this for any database that can store UTF-8 strings and has a sorted index for these doing string sort. The person in the talk wanted to store data about finite groups and large group orders are one such application.

ArangoDB starter is now bundled with the distribution and can start single server instances as well as cluster. It can use Docker or normal processes.

I think yes. The RocksDB engine is much better than the old engine in situations in which the stored data is larger than the available RAM. It all depends on the concrete queries issued but in general I would say: "Definitely yes!"

Max from ArangoDB here. Just to avoid disappointment: The official Docker image arangodb will need a few days to be updated and be validated by Docker. Use arangodb/arangodb:3.0.0 in the meantime.

One of the developers of ArangoDB here.

Let me explain this quotation. When your graph data (including indices) do no longer fit into the RAM of a single server, you can either live with the higher latency of loading data from disk or you can use sharding, which will lead to communication and therefore slower traversals.

That does not mean that things stop working, but performance will be less good, you can no longer visit tens of millions of nodes per second in a traversal as in RAM on a single server.

If you actually only traverse a much smaller hot subgraph, I would probably go for the disk based single server approach.

If your graph has a natural known clustering, then an optimized sharding solution with fine tuned sharding keys us probably your best bet.

You can do all this with ArangoDB.

However, graph traversals vary greatly in many respects, and your mileage may vary accordingly, with any approach.

I would love to chat in more detail about your use case.

Disclaimer: One of the ArangoDB developers here. As published in the white paper, here it is:

    ```{ "arangodb": {
        "async-replication": true,
        "nr-dbservers": 80,
        "nr-coordinators": 80,
        "framework-cpus": 0.5,
        "role": "arangodb",
        "principal": "pri",
        "minimal-resources-agent": "mem(*):512;cpus(*):0.5;disk(*):512",
        "minimal-resources-dbserver": "mem(*):8192;cpus(*):3;disk(*):8192",
        "minimal-resources-secondary": "mem(*):8192;cpus(*):1;disk(*):8192",
        "minimal-resources-coordinator": "mem(*):8192;cpus(*):3;disk(*):8192",
        "secondaries-with-dbservers": true,
        "docker-image": "arangodb/arangodb-mesos:devel"
      }
    }```

I only say that on many days I spend more time in my vim than in my bed. I usually run it in a terminal Window.

(Disclaimer: Max from ArangoDB) We have invested considerable effort to optimize each database. Obviously, we know our own product better than the others. However, we have asked people who know the other products better, and we keep this investigation open for everybody to contribute and to suggest improvements. As you can see from last week's post, there have been very good contributions, we have tried them out and have published the improved results.

(Disclaimer: Max from ArangoDB here) I am all for database pragmatism. Fortunately, the choice "graph database" or "non-graph database" is no longer binary. We at ArangoDB are convinced that graphs in data modelling have their merits (namely when you need "graphy" queries), but you do not want to be locked in to the graph data model. Therefore we argue for multi-model databases, which can give you graphs, but do not force you to use graphs for all and everything. With "graphy" I mean queries that involve paths in a graph whose length is not a priori known (e.g. ShortestPath).

No worries, yes, there is a lot of bad information about graph databases, to begin with, some seem to believe that everything out there can best described by a graph, which is clearly wrong. I have myself written something about this in this article: https://medium.com/@neunhoef/graphs-in-data-modeling-is-the-...

Furthermore, I am currently working on another article for the O'Reilly radar blog presenting a nice case study in which a multi-model database was very useful, because document queries and graph queries were both used extensively.

1B vertices and 100B edges will definitely be a challenge for any graph database and I find it highly likely that ArangoDB in its current version will not show a very good performance for a data set of this size. Obviously, it will always depend on the particular queries you need, and on whether the graph has a natural cluster structure that can be used for sharding.

(Disclaimer: Max from ArangoDB here)

The purpose of this benchmark series was not to provide a comprehensive test of all these databases. We only wanted to demonstrate that a multi-model database can successfully compete with specialised solutions like document stores and specialised graph databases.

I agree to your comment about graph databases, the crucial thing is that the neighbors of a vertex can be found in time proportional to their number, and that the queries involving an a priori unknown number of steps (graph traversals, path matching, shortest path, etc.) run efficiently in the database server and can be accessed conveniently from the query language.

I am surprised that nobody mentions the great performance of Google's V8 engine here...

For some additional thoughts about schemaless vs. schema-enforcing datastores see

  https://gist.github.com/neunhoef/9b6749089775e472d44c

Interesting article, did you consider any of the newer NoSQL solutions that offer a data model more similar to MongoDB but more, often configurable, consistency guarantees (like ArangoDB or RethinkDB)? I could imagine that the effort of a migration would have been considerably smaller...

Another good rule I tend to use is that if your queries will involve variable lengths of paths in your graph, it was probably a good idea to model using a graph. This is because another model would almost certainly need multiple joins, which can kill performance quite quickly.