At Quantcast we have physical servers in 14 cities. We use anycast to achieve site failovers in 6 seconds. Downtime for us would impact millions of websites, so we don't have downtime.
At Quantcast we have physical servers in 14 cities. We use anycast to achieve site failovers in 6 seconds. Downtime for us would impact millions of websites, so we don't have downtime.
(The trend for regular ISPs is probably improving, except that mobile/carrier DNS is often particularly broken. It would be interesting to do monthly surveys of this.)
It's nontrivial to determine exactly when to drop the announcement. And be careful, because if you are too eager to drop the announcement, you may do it in more than one site at a time.
At first we used DNS with short timeouts, but those timeouts are only advisory and are ignored by some implementations. We would see most traffic tail off within 10 minutes for a one minute timeout, but it took many hours for all the traffic to migrate over to the new DNS. The folklore on using less than one minute for a DNS timeout is that a huge percentage of implementations ignore sub-minute timeout. Funny how much of the Internet's operation is passed along as folklore and not really known for sure.
Thanks for asking. Hacker News should be about sharing best practices and making the Internet a more reliable place.