Hacker News Reader

Comment by tinymollusk | Hacker News Reader

But the larger issue -- that they have direct financial incentive to unconsciously bias the results -- still exists. This isn't anything against Arango's team, I'm sure they're lovely people, but even physics experimental results have been shown to be biased when the experimenter knew the testing hypothesis.

"I think I've been in the top 5% of my age cohort all my life in understanding the power of incentives, and all my life I've underestimated it. Never a year passes that I don't get some surprise that pushes my limit a little farther." -- Charlie Munger

View on HN

Performance tests - especially those of databases - are a very complex and resource-consuming venture. And because each use case is different, the benchmark published somewhere does not fit your specific problem and the available environment/budget.

Unfortunately, there is no independent organization that believes in this and defines scenarios that are tested in different environments.

Very good points. Seems like this leaves us with "trust benchmarks that probably aren't generalizable to your use-case" or "run your own expensive benchmarks before you have the scale to have the data to simulate the scale".

What's the best strategy here? I've defaulted to MySQL, pg, or sqlite, based on my preference of the moment. I haven't had enough good/bad outcomes to form a stronger opinion, but I'm now realizing I've never really thought this decision through (despite building a few data systems that I'm pretty proud of).

We just published an Update to the Benchmark, please find it here: https://news.ycombinator.com/item?id=16473117

Or Feynman: "The first principle is that you must not fool yourself and you are the easiest person to fool."

True, but so does everybody else. You shouldn't trust vendor benchmarks, but that doesn't mean you can't read them and use them as sort of a heuristic to help you come to your own conclusions. It might not be particularly scientific, but it's probably more than accurate enough for 99% of people.

True, but that is because every major database is good enough for 99% of people even when run on the cheapest linode available ;).

Reply on news.ycombinator.com