Hacker News Reader

Comment by janemanos | Hacker News Reader

In your 2015 article, you criticized that the ArangoDB team restarted the instances after each test run. In this 2018 edition, they don't do this anymore.

View on HN

You want to trust a vendor that used to restart the db between queries for their benchmarks?

Yes, because they learned. Arango has honest people that are open to criticism.

But the larger issue -- that they have direct financial incentive to unconsciously bias the results -- still exists. This isn't anything against Arango's team, I'm sure they're lovely people, but even physics experimental results have been shown to be biased when the experimenter knew the testing hypothesis.

"I think I've been in the top 5% of my age cohort all my life in understanding the power of incentives, and all my life I've underestimated it. Never a year passes that I don't get some surprise that pushes my limit a little farther." -- Charlie Munger

Or Feynman: "The first principle is that you must not fool yourself and you are the easiest person to fool."

True, but so does everybody else. You shouldn't trust vendor benchmarks, but that doesn't mean you can't read them and use them as sort of a heuristic to help you come to your own conclusions. It might not be particularly scientific, but it's probably more than accurate enough for 99% of people.

True, but that is because every major database is good enough for 99% of people even when run on the cheapest linode available ;).

Performance tests - especially those of databases - are a very complex and resource-consuming venture. And because each use case is different, the benchmark published somewhere does not fit your specific problem and the available environment/budget.

Unfortunately, there is no independent organization that believes in this and defines scenarios that are tested in different environments.

Very good points. Seems like this leaves us with "trust benchmarks that probably aren't generalizable to your use-case" or "run your own expensive benchmarks before you have the scale to have the data to simulate the scale".

What's the best strategy here? I've defaulted to MySQL, pg, or sqlite, based on my preference of the moment. I haven't had enough good/bad outcomes to form a stronger opinion, but I'm now realizing I've never really thought this decision through (despite building a few data systems that I'm pretty proud of).

We just published an Update to the Benchmark, please find it here: https://news.ycombinator.com/item?id=16473117

I trust but verify. When dealing with Arango I’m Not openly hostile to their claims. I’ve made several libraries for it even though I don’t use it. Contrast this with Mongo. Their web scale carpet bombing campaign makes me not use them at all. I immediately dismiss them. That is how I found Arango during my booklet writing days.

Reply on news.ycombinator.com