> When we benchmarked JDK 15, we saw that Java 15 was 11.24% faster than Java 11. Now, the gain of Java 17 over Java 11 is less. Does that mean that Java 17 is slower than Java 15?
> Well, no. Java 17 is faster than Java 15 too. Those previous benchmarks were run on a different codebase (OptaPlanner 7.44 instead of 8.10). Don’t compare apples and oranges.
Author is defending their other article by discrediting their own methods - what those percent values even mean if they differ so much between different versions of the same benchmark? Shouldn't benchmarks be profiled for specific workload types to simulate real life examples? How could you possibly change the workload so much it affects result so significantly and not change the nature of your benchmark?