Major Mixed Integer Linear Programming Solver Faked Performance Data

gurobi.com

14 pointswhatever14 comments

This title seems like an exaggeration of what is claimed in the article. In the article, they state that they benchmarked their solver in a biased way that made their solver look like it performed better than it did, not that they faked performance data altogether.

Far from exaggeration. In fact CPLEX and Express decided to go nuclear and remove their solvers from the benchmark set, so now we will be in the dark, regarding relative performance http://plato.asu.edu/bench.html

This is pretty sad. Unlike databases and other software, MIPS solvers have very good benchmarking standards, and though they don't cover every case, you can typically see which solvers are best for certain types of problems. Having met the Gurobi founders, they genuinely don't seem like they'd do something like this on purpose.

Now Mitellman's only benchmarks for this category will be the slower solvers.

I guess there will still be third-party benchmarks, in particular as part of the papers that go along the updates to the MIPLIB sets of benchmark instances?

Reply on news.ycombinator.com