At least, have it respond only to authenticated requests. Caddy supports client certificate authentication:
https://caddyserver.com/docs/json/admin/remote/access_contro...
https://caddyserver.com/docs/json/admin/remote/access_contro...
admin listen unix//var/run/caddy/admin.sock