I did, and contrary to unfounded meme, this was actually possible before PulseAudio was introduced.

Used JACK extensively before Pulseaudio came along .. never had any issue with it that Pulseaudio would solve.

Of course having audio switch when you switch applications has its benefits as well...