Hacker News Reader

Comment by jakewins | Hacker News Reader

Haha, I almost spit out my coffee when I saw how poorly we performed here! Reading the code, it's kind of incredible that Neo4j even competes - all the databases benchmarked are running with 25 concurrent database connections:

https://github.com/weinberger/nosql-tests/blob/master/arango...

Except Neo, which is given a single connection to work with :(

https://github.com/weinberger/nosql-tests/blob/master/neo4j/...

When I modify it locally to just set {maxConnectionPoolSize: 25}, like how the other databases are set up, I get roughly an order better performance. neighbors2 went from 0.0226ms to 0.0068ms, for instance. Single write sync went from 3.36415ms to 0.77549ms avg/op..

Benchmarking is hard!

View on HN

Hey Jake... thanks again for your feedback... we just published an update to the Benchmark following your feedback for Neo4j, please find it here: https://news.ycombinator.com/item?id=16473117

True, benchmarking is really not trivial. Thanks for pointing us to the bug in our benchmark suite and sorry for our mistake. We fixed it https://github.com/weinberger/nosql-tests/pull/24 and reran the benchmark. We will publish an update soon and also fix the stats in the original post. Indeed, Neo4j with connection pooling improved stats in a few test cases but the overall ranking is not really affected.

Thanks again for helping us to improve the benchmark and creating fair conditions for all products tested!

Jan from ArangoDB here:

Did you read the appendix?

"We used a TCP/IP connection pool of up to 25 connections, whenever the driver permitted this. All drivers seem to support this connection pooling, except Neo4j. We sent instead twenty-five requests via NodeJS to Neo4j."

Just to avoid unfair conditions now, if connection pooling is possible with the neojs driver, then you would have to configure the benchmark scripts to not send 25 requests via Node as well.

We tricked ourselves with such stuff when preparing the benchmark.

Did you read the manual where it says the driver permits connection pooling?

http://neo4j.com/docs/developer-manual/current/drivers/clien...

Hi Max.. thanks also for your feedback. We worked it into an update to the benchmark, please find it here: https://news.ycombinator.com/item?id=16473117

Yep, I removed the CONCURRENCY constant from the neo4j description.js, so the numbers above are for Neo running without that async.eachLimit call, just like the rest of the databases.

A single session object is backed by a single connection, so even though you "send" 25 requests via async.eachLimit, the result is that they are serialized over the single connection.

For future reference, I find the back ticks to decrease readability, at least on iOS.

Agree, I keep forgetting that those don't turn into monospace on HN. Changing that.

Reply on news.ycombinator.com