DNS is an awful way to do failover.
DNS is an awful way to do failover.
Route 53 supports DNS TTLs as low as 0 seconds. ELB and S3 endpoints both have 60 second TTLs. My experience with flipping names like www.amazon.com doesn't reflect the 15% figure. I've seen about 97% of web traffic honouring the TTL and flipping quickly. Within 5 minutes almost all of the rest too. We also take CloudFront sites in and out of service for maintenance, and in 5 years I've never anything like a 15% straggler effect.
That said, we do see a very small number of stragglers. While resolvers over-riding TTLs hasn't shown up as a significant problem, buggy clients can be; we come across clients now and then who either don't re-resolve ever (Various JVMs and their infinite caches are a common cause), or only re-resolve on failures (which is fine for failover, but not great for traffic management).
If you have a distribution time plot for the 15% figure it'd be interesting to see; https://www.dns-oarc.net/ would be a good venue, https://lists.dns-oarc.net/mailman/listinfo/dns-operations is the open list. Ignoring TTLs for a week is very concerning; it would very likely break many DNSSEC configurations. Is it possible you were dealing with robots?
But I ran across this around the year 2000. And the biggest offender was named AOL. I could well believe that it is very, very different today.
Typically rural ISPs or Alaskan ISPs (they don't want to pay to forward all the requests so they override the TTLs).
This is when we flip Netflix IPs.