Excellent point; it does appear that the author did consider timing attacks in at least one location:
https://github.com/mohammed90/caddy-ssh/blob/master/internal...
https://github.com/mohammed90/caddy-ssh/blob/master/internal...
The maintainers gliderlabs/ssh have plans for a new version with more ergonomic APIs but the work hasn't started yet. I didn't want to wait either.