ZigBee just seems way too unreliable. WiFi devices have - for me at least - been rock solid in terms of reliability, while ZigBee stuff randomly disappears or needs to be relinked all the time (and it is hard to debug when this happens, unlike just pinging an IP on your LAN)
Home assistant et al are typically overkill and another point of unreliability. Just run Mosquitto (MQTT broker) locally and run a local NodeRed instance if you need anything sophisticated in terms of logic.
Since you are using MQTT it is trivial to write your own app for controlling everything from your phone. Use tailscale if you need to do things when off-LAN.