There are 2 problems for me: 1) I cannot tell by feel what to press to quickly mute, jump back 10 seconds, pause, etc. I have to look at the phone.
2) When I look at the phone, my phone has locked. So I have to wait a second for face-id (or enter a passcode if its too dark).
It's a shame to see how the IR blaster has disappeared from phones, just like the FM radio functionality that the SoC probably still has support for. Only weird, cheap, and very Chinese phones still seem to sport these features.
You're also often limited to the app provided by the manufacturer of the 3.5mm dongle, which isn't always great.
The other path is with the chromecast, which turns on with the display.
Did you ever notice that Playstations do audio much better than Chromecast? So I rarely use my Chromecast. The several hundred dollar difference is not an argument to me, I expected the Chromecast to not have a difference in audio.