Comment by mholt | Hacker News Reader

Comment by mholt | Hacker News Reader

This is a great writeup overall. I was happy to see Tyler's initial outreach before conducting his tests [0]. However, please note that these tests are also being revised shortly after some brief feedback [1]:

- The sendfile tests at the end actually didn't use sendfile, so expect much greater performance there.

- All the Caddy tests had metrics enabled, which are known[2] to be quite slow currently. Nginx does not emit metrics in its configuration, so in that sense the tests are a bit uneven. From my own tests, when I remove metrics code, Caddy is 10-20% faster. (We're working on addressing that [3].)

- The tests in this article did not tune reverse proxy buffers, which are 4KB by default. I was able to see moderate performance improvements (depending on the size of payload) by reducing the buffer size to 1KB and 2KB.

I want to thank Tyler for his considerate and careful approach, and for all the effort put into this!

[0]: https://caddy.community/t/seeking-performance-suggestions-fo...

[1]: https://twitter.com/mholt6/status/1570442275339239424 (thread)

[2]: https://github.com/caddyserver/caddy/issues/4644

[3]: https://github.com/caddyserver/caddy/pull/5042

Thanks, Matt! I've pushed the revised section measuring sendfile and metrics changes, so those should be accurate now.

Phew. Caches are purged, my errors are fixed. I can rest finally. If folks have questions about anything, I'm happy to answer.

Just want to say your writing is the best quirky balance of fun and substance, reminiscent of Corey Quinn [1]. Thanks for doing a so damn much work and for the instantly relatable phrase "Nobody is allowed to make mistakes on the Internet".

[1] https://www.lastweekinaws.com

Thank you, that's very kind! There's a reason I included Corey's name in my hastily cobbled-together skeleton meme [1]. Hopefully my writing achieves that level of technical approachability.

[1]: https://blog.tjll.net/reverse-proxy-hot-dog-eating-contest-c...

How did I miss that. Anyway, you succeeded.

That’s very kind of you to say. Do I get to put this on my resume?

THE MAN HIMSELF

I can die happy

> - All the tests had metrics enabled, which are known[1] to be quite slow. From my own tests, when I remove metrics code, Caddy is 10-20% faster.

But disabling metrics is not supported in standard Caddy, you need to remove specirc code and recompile to disable it.

So maybe benchmarking with it isn't fair to Nginx?

> But disabling metrics is not supported in standard Caddy, you need to remove specirc code and recompile to disable it.

We're addressing that quite soon. Unfortunately the original contributor of the feature has been too busy lately to work on it, so we might just have to go the simple route and make it opt-in instead. Expect to see a way to toggle metrics soon!

Update: PR opened: https://github.com/caddyserver/caddy/pull/5042

That was fast. I love it!

There's always going to be some cost to metrics, going forward you probably just want to document it and then update the figure as you tune it. Higher performance opt-in metrics are the sort of thing a company using it at scale ought to be able to help with/sponsor work on.

Absolutely. The plan is to make it opt-in for now, and then having a company sponsor the performance tuning would be very welcomed. Otherwise it'll probably sit until someone with the right skills/know-how and time comes along.

Yeah, I think Fair comparisons are:

* How do these things perform by default. This is how they're going to perform for many users, because if it's adequate nobody will tune them, why bother.

* How do these things perform with performance configuration as often recommended online. This is how they'll perform for people who think they need performance but don't tune or don't know how to tune, this might be worse than default but that's actually useful information.

* How do these things perform when their authors get to tune them for our test workload. This is how they'll perform for users who squeeze every drop and can afford to get somebody to do real work to facilitate, possibly even hiring the same authors to do it.

In some cases I would also really want to see:

* How do these things perform with recommended security. A benchmark mode with great scores but lousy security can promote a race to the bottom where everybody ships insecure garbage by default then has a mode which is never measured and has lousy performance yet is mandatory if you don't think Hunter2 is a great password.

> How do these things perform by default.

Agreed on this one -- today I'm looking at how to disable metrics by default and make them opt-in. At least until the performance regression can be addressed.

Update: PR opened: https://github.com/caddyserver/caddy/pull/5042 - hoping to land that before 2.6.

Good stuff dude. Listens to users, sees a problem, doesn't take it personally, makes a fix. Caddy's going places.

I'm also grateful that Dave, the original contributor of the metrics feature, isn't taking it personally. We love the functionality! Just gotta refine it...

> All the Caddy tests had metrics enabled

One of the great mysteries in (my) life is why people think that measuring things is free. It always slows things down a bit and the more precisely you try to measure speed, the slower things go.

I just finished reducing the telemetry overhead for our app by a bit more than half, by cleaning up data handling. Now it's ~5% of response time instead of 10%. I could probably halve that again if I could sort out some stupidity in the configuration logic, but that still leaves around 2-3% for intrinsic complexity instead of accidental.

Reply on news.ycombinator.com