> Failover needs to happen within a few seconds, and most importantly, while the client is still connected to the load balancer. You can't do this using DNS with short TTL values.
No it doesn't and it's good enough. Humans are not stupid, they will try again, web browsers will try other IPs too. For custom clients it's not even a question, as they can implement transparent failover in the client itself. Essentially this is all about web browsers.
The big problem with anycast routing is that you still have to rely on a single AS/company and have a huge SPOF, that will fail from time to time. A few hours of downtime every other year is as good as you can get, less than that would be pretty much impossible for such architecture. So, if you need anything better, you would have no choice but implement a DNS failover.