Some sort of USB firewall? Something you can share?
* https://jdebp.uk/Softwares/nosh/guide/user-virtual-terminal-...
* https://jdebp.uk/Softwares/nosh/guide/commands/user-vt-reali...
This is for a virtual terminal system. For X11 or Wayland, one would have to replicate the same idea in an appropriate form, and stop using the multiplexed devices.