I feel similarly. For me it’s less about my unique customizations and more about this paranoia of there being something remotely sensitive in my ssh configs or something… the idea of hostnames, ips, domains, etc “leaking” worries me.
I use chezmoi to manage my dotfiles, if anyone has any advice on how to handle these worries I am all ears. I would love to share mine, even to just be able to point coworkers at my config.