The attack surface of Vim is admittedly small, but the browser is a much bigger target. I want my browser to download files, upload files that I provide, and various other things on an allow list, but not more.
IDEs, the tools invoked by them, and their build artifacts should have access to the project assets and not much more.
There are certainly applications that require more privileges, but I'm sure we can come up with ways to restrict them as well. The problem is that it's quite late to retrofit it onto desktop Linux.
Containers are not designed to be security boundaries. They isolate their contents from the host system, but not primarily because of security. They can be used for security, but certainly not with default settings.