I forward the pulseaudio socket (pipewire-pulse, to be precise) over ssh and it works fine.
ssh -R /run/user/1000/pulse/native:/run/user/1000/pulse/native remote_host ...
assuming the SSH client is the one with the audio device and remote_host is the one that wants to play audio. If it's the other way around then replace -R with -L.