When VSCode connects to the remote using the plugin, it installs an entire VSCode server - seamlessly to the user every time they connect to a remote - that keeps track of all project facilities, shells, extension accessory runtimes, takes care of embedded or computationally heavy tasks like compiling, building, running project-wide code analysis tools, etc. while keeping all settings, editor windows, and accessory panes local (which is critical for UX and latency). VSCode appropriately partitions responsibilities between the local and the remote, automatically restores IDE infrastructure on the remote as needed, and enforces the partitioning architecturally for all extensions.
This architecture is not an accident - it's rooted in VSCode's origin as a browser-based IDE, and makes use of the LSP and other features that don't exist in Emacs/TRAMP because nobody really thought deliberately about running the extensions at an arm's length from the editor UI in Emacs, using an async protocol that doesn't allow extensions to impact core UI latency. But the flipside is that the "client" part I described above does place a lot of trust in the "server" part, just as you might expect a browser-based application to do.
Whether you consider this architecture bananas or not depends on your security model. It's probably not the best idea to let people SSH to production using this plugin. If you are trying to rely on it to partition a novel AI tool away from part of your infrastructure, yeah, it wasn't really meant for that. The plugin predates agentic AIs by 5+ years. But it's an incredibly powerful and useful feature. I don't think taking cheap shots against it is helpful when the actual bananas thing is trying to use it in a way it was not intended for.