The entire point of the benchmark is virtual calls vs templated static calls. Everything else is what should be controlled.
If virtual calls are used essentially, then it's a canoe-versus-bicycle comparison, because everything else cannot be controlled. The program based on static calls has to be written quite differently to solve the same problem, and the benchmark then measures the entire approach. Plus the benchmark doesn't account for benefits that it doesn't measure, like maintainability and extensibility of the code.
Or should we only ever benchmark the entire software stack at once, even though people do in fact sometimes use CRTP and virtual functions in the same situations?