You configure Caddy to disable certain challenge types[0], which in this is the HTTP challenge, like so:
example.com {
tls {
issuer acme {
disable_http_challenge
}
}
file_server
}