But disabling metrics is not supported in standard Caddy, you need to remove specirc code and recompile to disable it.
So maybe benchmarking with it isn't fair to Nginx?
* How do these things perform by default. This is how they're going to perform for many users, because if it's adequate nobody will tune them, why bother.
* How do these things perform with performance configuration as often recommended online. This is how they'll perform for people who think they need performance but don't tune or don't know how to tune, this might be worse than default but that's actually useful information.
* How do these things perform when their authors get to tune them for our test workload. This is how they'll perform for users who squeeze every drop and can afford to get somebody to do real work to facilitate, possibly even hiring the same authors to do it.
In some cases I would also really want to see:
* How do these things perform with recommended security. A benchmark mode with great scores but lousy security can promote a race to the bottom where everybody ships insecure garbage by default then has a mode which is never measured and has lousy performance yet is mandatory if you don't think Hunter2 is a great password.
Agreed on this one -- today I'm looking at how to disable metrics by default and make them opt-in. At least until the performance regression can be addressed.
Update: PR opened: https://github.com/caddyserver/caddy/pull/5042 - hoping to land that before 2.6.
We're addressing that quite soon. Unfortunately the original contributor of the feature has been too busy lately to work on it, so we might just have to go the simple route and make it opt-in instead. Expect to see a way to toggle metrics soon!
Update: PR opened: https://github.com/caddyserver/caddy/pull/5042