I might sound naive, but as a true hyperscale internet company to plan for disaster scenario like this as a consumer of fastly?
How could you plan for an outage like this by fastly and how could you mitigate this?
How could you plan for an outage like this by fastly and how could you mitigate this?
I was thinking more about this though and it has its own problems. You want a short TTL so failover is fast, but this increases the number of DNS lookups people have to do (and DNS lookups can be very slow!).
Additionally, a short TTL means you're more vulnerable to problems like the dyndns attack [1] from 2016: names with longer TTLs were up for longer since they preserved the correct DNS records for longer.
But if you have a long TTL, even if you fail over, you'll still be down for at least as long as the DNS TTL pointing to the bad CDN.
Maybe, you could do DNS roundrobin against multiple CDN providers at once. Say you used 4, then if one went down, only 25% of requests would fail, and you could just remove the failing entry. This seems very expensive!
Honestly, the cost of these solutions is probably not worth it. The product I work on went partially down during the fastly outage. Then it came back up and everything is back to normal. It really won't impact us much at all. Shrug.
[1] https://twitter.com/RichardTowers/status/1402312111078903812
[2] https://twitter.com/RichardTowers/status/1402312308077076484