Here's one: it does not support dynamically loadable modules, like most (all?) Go programs. So if you need e.g. geoip, you have to build your own, and then maintain it, tracking CVEs, etc. You can't rely on your distribution's package maintainer to do the work.
For example to use rate limiting I just have a Dockerfile like this:
FROM caddy:2.9.1-builder AS builder
RUN xcaddy build --with github.com/mholt/caddy-ratelimit
FROM caddy:2.9.1
COPY --from=builder /usr/bin/caddy /usr/bin/caddy
Only a POC, supported in Linux and macOS, and basically relies on doing casts from loaded symbols into what they are supposed to mean.
I need route 53 and a few other DNS providers built in for let's encrypt support and the docs implied that I was going to have to build those plugins myself?!!!
I stopped reading at that point because cert bot is trivial to install and just works with the web server that was also one command to install. At no point did I have to create a ephemeral container just to build nginx or certbot...