Hacker News Reader

Comment by cwyers | Hacker News Reader

I really, really distrust these kinds of evaluations when they come from someone whose product is included in the comparison. Even if everything is above-board, they're not going to publish if it shows their product just completely sucks at it. That kind of publication bias makes these kinds of results a lot less trustworthy than independent benchmarks even if you assume the best of intentions from the people putting them out there.

View on HN

Ingo from ArangoDB here. I agree that vendor tests are always biased, of course you want to show that your product is competitive.

But as there is no independent institution that compared our product and as we want to know where we stand with ArangoDB, Claudius did his own tests. And as the work is already done, why not share it.

We tried our best to do it as open as possible. PostgreSQL performed very well and we have a problem with memory consumption - have a look at the charts, we will try to improve there.

- Every database configuration is public

- All test scripts are available on Github

- We publish updates if we get pull-requests or comments with suggestions for improvements

We did that before and after the last test, some database vendors sent us improved snapshots of their databases which found their way into the latest products (OrientDB and Neo4j).

If you have suggestions for improvements, please let us know.

> PostgreSQL performed very well

Despite the fact that you crippled it by not using jsonb columns.

Why so hostile? Personally, I assume this is just an instance of them not being PostgeSQL experts. So, before jumping to conclusions (namely that they deliberatly skewed their test, despite being very open in the way they described the setup), I'll instead wait and see -- perhaps they'll explain why they used json instead of jsonb, or perhaps (better!) they can updat the test to also include jsonb.

> I assume this is just an instance of them not being PostgeSQL experts.

this illuminates the need for "experts" in the different components in the stack. it's intellectually dishonest to claim that a technology is not up to the task when it's not been properly treated in the first place.

If they're not Postgres experts then they shouldn't be including Postgres in the line-up. Or ask someone that ask someone that knows better.

That's why they welcome pull-requests.

They wanted to play with the other kids, but no one invited them, so they threw their own party despite not knowing how to throw one.

Biggest problem of (small) German tech companies IMO.

I worked for one myself and they had really good core technology, but that was it. The angelo-saxon companies always outplayed them, because they're just so much better at PR. Even their developers are better at this than most marketeers that burn money on a daily basis in Germany...

I think his hostility comes from the fact that if you google just "postgresql json" the second hit is the official documentation that goes over jsonb right in the second paragraph.

http://www.postgresql.org/docs/9.4/static/datatype-json.html

You can also see that the very first link outside of the official documentation:

https://www.compose.io/articles/is-postgresql-your-next-json...

Explains the benefits of jsonb and how to use indexes.

If you're going to include a database, at the very least do 20 minutes of research on it. If that can't be given, then just don't include something.

That's like saying "I've just installed MongoDB and read the intro documentation page, I'm now going to benchmark it against a MySQL cluster, which I have years of experience with, and that I helped develop." (E.g.: they developed ArangoDB, so they should be experts in at least that, right?)

> Why so hostile? Personally, I assume this is just an instance of them not being PostgeSQL experts.

Postgres' official documentation on json and jsonb is rather concise: http://www.postgresql.org/docs/9.4/static/datatype-json.html

The difference between the types is described in paragraph two. So, they either benchmarked Postgres while having no idea whatsoever what they were doing, or they were deliberately crippling the competition.

Neither option is confidence inspiring.

(And the first option seems sketchy, seeing how they then went and re-created the whole benchmark as classical RDBMS setup for the second postgres test.)

If they weren't going jsonb and the gin jsonb-path-opts index, they shouldn't have even included postgres.

The particular benchmark seems bullshit, too. For postgres they seem to intentionally use the less performant json column type instead of jsonb.

Not that I can verify it, because the code in the linked public "No magic, no tricks – check the code and make your own tests!" repository doesn't match the published results and doesn't even work at all with postgres…

EDIT: Okay, they pushed a new version containing the Postgres data now. They ARE using the cripplingly slow json columns, not jsonb columns recommended by the documentation.

Just to be a pedant, the JSONB format as I understand is marginally slower at inserts and orders of magnitude faster at everything else...

It's fractionally slower, true, because of the serialization hit (string to binary). The real juice comes from the GIN index - and if you apply it to specific columns instead of a complete document, you have a rocket ship on read.

> It's fractionally slower, true, because of the serialization hit (string to binary).

And with JSON columns you have to serialize on accesses, which is a lot slower in the read-mostly tests.

Stupid question: isn't that the typical result of indexing? Slower writes for (much) faster lookups?

And despite that Postgres destroys all the other solutions at everything other than the non-sync (lol) single write case and graph traversal.

If anything it just proves even after almost a decade of these "NoSQL" solutions being around they still can't compete even on basic queries with Postgres which is a fairly conservative SQL solution.

I think this is a classical case of "use the right tool for the right purpose". If you want to do "classic database" stuff, use a classic SQL database. But if e.g. graph traversal is a cruicial operation for your application, then looking at NoSQL solution seems to be quite interesting.

In other words: I wouldn't call a screw driver a bad tool, just because it's not as good at driving nails into wood as a hammer.

I think the issue is how many NoSQL solutions over recent years have billed themselves as the default start-with-us data store, and SQL as the more complicated niche product...when it's really the other way around.

this is more constructive that flooding the communications channels with hidden facts like Neo4j does.

WTF with VCs investing into all this crap? There are very few scenarios were you would be better off with NoSQL solution and there are established players serving those niches already.

Reply on news.ycombinator.com