I've taken this approach halfway and only install plugins/config that I can easily do without. Things like git-gutter [0] and a line at the 80th column [1] are great, but I don't depend on them when I log into a remote machine. If I really want some features, I have them somewhat organized in my .vimrc [2]. It's a great compromise; I find that I haven't thought about major editor configuration in a couple of years.
[0] https://github.com/airblade/vim-gitgutter
[1] set colorcolumn=80 textwidth=79
[2] https://github.com/stefco/dotfiles/blob/master/linkfiles/.vi...
[edit] Just want to add that I 100% agree that erring towards vanilla vim is a great idea; I used to use emacs and switched to vim precisely because the starting point fit my use style far better than vanilla emacs.