I wonder how your setup is different than mine. I also switched to Pipewire (some months ago) and I'm using NixOS and Arch Linux, and on both I could just change the audio output to my speakers on my monitors (via HDMI) with the usual device switch, same as before with just PulseAudio. Everything works perfectly.