I wished this too, and got close enough by buying the Hue remote switches:

https://www.philips-hue.com/en-us/p/hue-dimmer-switch--lates...

It turns out there are really only 3-4 different temperatures I use normally, and this allows you to switch between them. I also use the Hue timers so I frankly don't need to use this that often, but it's nice to be able to switch between both scenes and brightness levels without pulling out my phone (edit: and more importantly so people who aren't me can control the lights if they need).

It's not a perfect solution: there are still some annoyances with people forgetting to leave the main switch on and only use the mounted remote, but overall I am very happy with this system.