I don't really see how the XZ backdoor is relevant to this conversation, since these types of exploits are the fear of server operators, and generally not people who run user programs like Emacs on their desktop. A rogue Elisp function could delete all of the files in your home directory if it wanted to, but people don't really complain about that because being able to delete files is a desirable function.
If you ask me, Emacs is a program where user freedom is and should be the ultimate goal. Adding security features that the user has to jump around 'for his own good' and only really serve as idiot nets are dubiously effective and only hinder this freedom. If someone wants to install a package from a source other than the official archives, then the trust is placed in him to actually look at the source code he's received. Even if you install an official package you should still look at the code, as it's quite useful to know what's going on inside it when you want to configure it or add functionality. The vast majority of packages I use are a single file and tend to not take up more than a few screens, so they are easily understandable.