I'm not sure it's even accurate to say that Wayland closes one of the holes when the hole Wayland closes isn't part of the system's security boundaries. It's like installing a deadbolt in a door standing in the middle of a room.
The problem with it is, that it works fine when one purely speak of being able to write and read from files, but the moment servers such as display servers or Pulsaudio and DBus come into play, the picture becomes more difficult.
All of those technologies work on a simple binary level where the subuser has access to the socket, or it does not, for finer grained control the kernel is required to speak the protocol, which will obviously not happen.
So, those services themselves must come with a means to filter communication appropriately, and sandboxing technology is beholden to the extent thereof.
Flatpak had to provide an alternative DBus-proxy server to do this with DBus, similar to using nested X11 servers; no solution has been reached for Pulseaudio, whereof I know, and no plans even exist for a variety of more obscure servers that software might need to communicate.
For instance, ZNC can be instructed from the IRC client to load modules that contain arbitrary code. Therefore, any IRC client that has acces to ZNC has ZNC's full capabilities, as it does not come with such finer granulation as of this moment.
If any sandboxing technology is to be effective, a large number of servers that are in common use often need to provide specific support for that specific sandboxing technology, or simply not be accessible at all from within it.
Pulseaudio is not getting much work these days, I believe the work currently is happening in Pipewire, which was built to have a fine-grained permission system that can work with any sandbox and is backwards-compatible with Pulseaudio.
I'm not sure how your IRC bouncer would work there, but presumably if you sandboxed that, it only needs to talk to the IRC socket and nothing else. For other obscure servers that have no concept of security, I'm not sure what can be done about that if nobody wants to modify them or replace them. You might just have to accept that they will need to run at elevated privileges and can clobber your system or home directory.