This was my experience as well. I really tried to use the vim plugin for vscode but there were too many behavior differences between it and native vim. And VSCode just has too many features that I've grown to love that I just find myself learning the native VSCode bindings and using those most of the time now.

I miss VIM, I really do, but I really only use it these days if I need to ad-hoc modify an individual file while I'm in a terminal somewhere.