Hacker News Reader

Comment by t1m | Hacker News Reader

Then again, MemSQL didn't design the benchmark. They beat Oracle at their own benchmark on massively cheaper nodes.

That doesn't mean a whole lot. Back in the early '00s, ATI and Nvidia were caught optimizing for certain benchmarks that were commonly used to compare video card performance (IIRC one of them was caught with code that specifically cut corners if it detected being run by the benchmark program).

I'm not saying MemSQL is doing anything nefarious, but the fact that the benchmark tests are controlled by someone else doesn't carry a lot of weight.

I would say that doing the other guy's benchmark is the most objective thing you can do. You should take it one step further and include other competitive products. In this case, it's pretty easy to beat the crap out of Oracle, it's an elephant. I would challenge the MemDB guys to run the same benchmark against VoltDB and report on that.

Also, let's get down to brass tacks - I don't see the actual queries that are being graphed in the MemDB post, and what is that graph actually measuring?

I think the LMDB team has done a really good job with this type of comparative benchmark. They took the LevelDB benches, then ran them against a lot of competitive products, both new and legacy, and gave a really good view of the entire field, in terms of performance on a number of dimensions:

http://symas.com/mdb/inmem/

http://symas.com/mdb/inmem/scaling.html

Reply on news.ycombinator.com