It is for this reason that my lights (Hue) and switches (Inovelli) are bound in ZigBee groups. HA adds some niceties, but the lights still work if it’s down. Accent lights I have bound to the Hue Bridge, because their app adds features HA doesn’t have, and they aren’t required for emergency egress.
The only gap I have, I think, is that because I’ve set all bulbs to respond to power restoration after a cut in last known state, if everything somehow lost its Zigbee binding, putting switches into dumb mode locally still wouldn’t help. However, since that’s an improbable scenario, I accept the remote possibility in exchange for not being awakened from lights coming on in the middle of the night. Plus I keep flashlights / headlamps throughout the house.