I personally use no plugins [1]. Zero friction moving from host to host. Vim is a great text editor, not a great IDE. For anything more sophisticated I prefer just switching to something like VS Code (with Vim bindings of course).
In any event, I just mitigate this problem by not overriding a bunch of standard behaviour. Also, none of the many plugins I use modify any standard behaviour in any serious way, they just help speed up certain actions when working in a large code base. There is very minimal context switch for me when editing a file remotely.
Nowadays I have a vimrc that just tweaks vanilla features the way I like, plus some highlighting for certain keywords regarding a C codebase I occasionally work on. The only plugin I use is a modified version of DetectIndent.
I've found that setting a custom colorscheme in vim averts this problem completely. If I see my colorscheme, my muscle memory is primed for my config. If I see the default vim colorscheme, my muscle memory reverts to default vim.
If somebody gave me a copy of otherwise vanilla vim to use that had my colorscheme, that would certainly throw me for a loop. But so far nobody has been that diabolical.
Tbh, the only thing that gets in my way are the exact tab-related settings you also have.