The only downside is that the difference between extensions installed on the remote and locally is a little confusing, but the extension ecosystem in VS Code has satisfied all of my use cases. I've also run into some occasional ssh hangups on weak connections, but I haven't experienced that isuse in a few months.
For years I was a linux guy, and now I see no reason to go back, because I can just remotely access a linux environment from whatever system I'm already on. Less time spent swapping between operating systems to work on projects, and a consistent environment, are both huge features.