And they’re always frustratingly incomplete and/or buggy. Some motions will be unimplemented, or will behave incorrectly, especially with regards to boundaries around whitespace inclusion or exclusion in a motion. Things you added to Vim and got used to are unavailable, and so your ysa") to put parentheses around the string your cursor is on just doesn’t work. (Lack of vim-surround definitely gets me frequently.)
In VS Code, it maintains its own undo stack, and the interactions between the two undo stacks are terrible and terribly frustrating—yet doing without might be even worse, because they behave very differently.
And too often in a browser context, ^W will close your tab, losing data, rather then being inhibited and erasing the last word, as it should. That one is practically unforgivable in a Vim mode, yet common. That, more than anything else, will make me avoid a Vim mode.