my biggest issue with agents in neovim or emacs is that I also use emacs and neovim to open or edit sensitive data (like ssh keys, etc) that I don't want to upload to a random LLM.
A quick solution I devised is to use bubblewrap to get a fully separate instance of nvim. Something along the lines of
alias lvim "bwrap --bind / / --bind $HOME/.config/{lazy,n}vim --bind $HOME/.local/share/{lazy,n}vim --proc /proc --dev /dev nvim"
works great (note: fish alias)