Benchmarking: Do it right or don't do it at all

The only thing i've ever gotten with mongo is a bunch of marketing BS honestly. We have worked with some of their sales people and technical people to try and figure solutions out. It was 100% marketing BS. no thanks for me, best of luck for you guys.

In any case, I was writing this to note that OnGres has replied to Mongo's reply setting an example of how tech discussions should happen: without derogatory and arrogant comments, open to valid criticism (i.e. with something more than words and numbers that cannot be reproduced) and transparency.

Check it out: https://ongres.com/blog/benchmarking-do-it-with-transparency...

In there you'll see how Mongo consistently mis-interpreted (or mis-represented?) the results. They kept mixing the benchmarks and constantly talked about an experimental driver and missing connection pooling. In fact, they did use the official Mongo Lua driver and the official Java driver for different benchmarks and they did some of the benchmarks with and without connection pooling and published both results.

It's really sad to see Mongo reply to a thorough benchmark like this. It probably has its flaws but instead of correcting them or publishing a better benchmark like the one they did (to magically get 240x...) they chose to mischaracterize the work of others, spreading FUD and accusing them of cheating and being dishonest.

Hopefully they'll turn around and fix it. All it takes is to publish how they got they amazing numbers so that others can comment, repro or dispute the benchmark.

However their analysis does seem fair that this benchmark was very biased.

As they say themselves: "They should then make those benchmarks reproducible and publish their results in full."