Hacker News Reader

Comment by yjftsjthsd-h | Hacker News Reader

Once upon a time, I worked in a saas company that would sometimes switch customers to a new instance of a service by switching DNS records -

1. Create instance of service running version n+1

2. Switch public DNS records to point to new servers

3. Wait for TTL to expire

4. Turn off old servers

(Obviously I'm simplifying; if nothing else there should be testing steps in there)

Unless I've missed something, wouldn't the author's suggestion to artificially raise the TTL by ignoring the upstream TTL result in the application breaking for customers if they used a DNS resolver that did this?

View on HN

Yes, for 40 minutes to 1 hour.

But I bet you still ran forwarders on the old hosts for at least an hour after you cut over DNS.

> But I bet you still ran forwarders on the old hosts for at least an hour after you cut over DNS.

I promise you we did not.

That's a surprise - I've handled migrations like this in the past, and we always setup a simple proxy to forward traffic for a while.

I've definitely lost count of the number of clients that would cache the old IPs, despite valid and low TTLs being in-place well in advance of a migration.

The impression I got from the senior sysadmins was that we considered clients caching records beyond TTL to be a bug on their side and not our problem, and (importantly) the nature of our business/clients allowed us to make that determination and not take corrective measures to compensate for client-side misconfigurations. As such, practicing traffic would have been considered at best unnecessary work (and at worst comprising our testing process and encouraging bad behavior).

I can appreciate that, I know that I would see traffic hit the old IP for >3 days. I suspect old Java clients, etc, that would resolve IPs once on startup and never again.

In our case it was worth keeping things working for a few days, but after a week at the outside we'd kill the proxying/forwarding.

There were major providers that at one point in time overrode TTLs to a minimum week if under 24h.

Reply on news.ycombinator.com