This may be tinfoil, but this is the kinda configuration that I want version controlled, and I've never felt good about adding ~/.ssh/config (or anything within ~/.ssh) to my dotfiles because I definitely do not want its siblings in version control. And I don't trust myself not to bone myself.
Instead I have some files elsewhere that I source into my terminal on start, containing:
export M4MINI=192.168.1.204