I've been doing this for a while now, but my file is now huge and it's cumbersome to edit. Is there no utility to mange that file?
alias compile-ssh-config='echo -n > ~/.ssh/config && cat ~/.ssh/*.config > ~/.ssh/config'
alias ssh='compile-ssh-config && ssh'
Compiles all your ~/.ssh/*.config files into a single ssh config file. It's simple and stupid and seems to do the trick.