Agreed. A better solution is to have all nameservers always return 2 A records in response to queries for a specific domain name. That way the browser will itself be responsible for trying to connect to each of the IPs to determine which one is alive (then with DNS pinning it will stick to the working one). And it does not matter if an intermediate DNS caching resolver ignores the TTL.
Of course the ultimate solution for HA is to implement anycast with BGP...