Agreed. Tried to reconcile this for some of my projects a while back and spent much more time I anticipated. This is an unexpected rabbit hole, between type casting, scoping of flags to subcommands and flags > configuration file > environment variables > default values precedence. And TOML / YAML / JSON / XML / INI multi-format support.
I made my hacks open-source, so if you have a Click-based Python CLI, try click-extra, contribute back, and help me solves this problem once and for all: https://kdeldycke.github.io/click-extra/config.html