This is what HDMI CEC[1] is for, right?
The devices are supposed to coordinate on which device will output audio (TV, sound system, etc.), and they are supposed to send control signals (volume up, volume down, and mute button presses) in a channel over the HDMI connection so that whatever device is playing audio can adjust its volume.
Of course this assumes all devices involved implement CEC (which is optional) and that the implementations aren't full of bugs. But the point is HDMI does have a solution for this in theory.
CEC is how I can punch the volume control on my Samsung TV remote and the volume on my Denon AV receiver changes. (And it's not the TV remote transmitting IR to the receiver. Within the first 5-10 seconds of powering on, pushing volume up on the TV remote makes the TV try to adjust its own volume even though it's not playing audio. Then it comes to its senses and realizes it should be forwarding that via CEC.)
---
[1] https://en.wikipedia.org/wiki/Consumer_Electronics_Control