Besides, if I just wanted to use my smartphone I would simply stick to my chrome cast devices. If I’m gonna be using the phone to control my entertainment center then the chrome cast provides by far the best interface because you have access to the entire app.
To be clear I use my TVs remote with HDMI CEC, not that insane apple remote.
That said, I just tried it and it does not seem to work for me in my setup. I just have a Samsung TV with the Apple TV connected. No other "AV system". The volume buttons do not seem to do anything with the TV. Now that I know it works in some situations I will start digging around.
The Apple TV remote controls the volume of the TV fine but I seem to recall that I programmed it for the power and volume of my TV when I set it up.
I'm not sure what remote your are using, but the one built into the iPhones pull down menu is instant for me.
Unfortunately I can't find any definitive source on the internet to confirm exactly what's going on.
Mine is behind the tv.
I have multiple of both Apple TV and Chromecast and they both have their benefits. The babysitter can control the Apple TV with their phone. With the Chromecast they need their own Netflix/Prime. Not nearly as good. For my own personal use Chromecast is best.
Edit: and of course on my newer TV’s most services are available as apps.
You may want to double check that statement and reset your "Apple Advertising ID".
Our family does this only because the actual remote is such a pile of crap.