Caddy also supports Unix sockets, which should be rather more difficult to smuggle requests to, and can be protected by file permissions:
admin listen unix//var/run/caddy/admin.sock admin listen unix//var/run/caddy/admin.sock