A data point: I'm a plugin-heavy neovim user, and I wholeheartedly testify that familiarity with vi modal editing is incredibly useful when ssh-ing into remote systems, and I don't have my plugins.
It does sound to me like that is not a consequence of the GUI-ness of VSCode, but just the man-hours that go into making it. In other words, it doesn't sound like that experience can't be replicated in terminal-only editors.
If anyone has experience with remote editing in a terminal editor, but not through an ssh shell (ie, the traditional way), I'm very interested in hearing your experience.
As usual, emacs has a mode for everything:
The mostly-single-threaded nature of Emacs gets in the way sometimes.