Why DNS-based Global Server Load Balancing does not work [2004] | Hacker News Reader

Why DNS-based Global Server Load Balancing does not work [2004]

tenereillo.com

21 pointsIgorPartola31 comments

If memory serves, Netflix users discovered that Akamai is using this strategy and everyone coming in from Google's DNS or OpenDNS was getting put into the same 'pool', and overloading the pipes in that pool. The recommendation was to use your ISP's DNS at least for your Netflix devices.

Or have I misunderstood?

(Post should probably have had the date listed; just realized this article is 7 years old. That doesn't make it wrong or bad, mind you...)

Fixed the title. The sad thing is that not much has been invented since then to aid with global load balancing.

http://tools.ietf.org/html/draft-vandergaast-edns-client-sub... is an RFC to fix this problem.

This isn't true for Google. Akamai has all kinds of archaic network issues that cause problems, but it isn't often OpenDNS related. They certainly do their best, and know our network map well as we send it to them whenever it changes.

For Google -- If you use OpenDNS, you will always go to the most appropriate Google datacenter to you (not OpenDNS).

THIS. Akamai is using technology designed in 1998. I wish they would work with opendns to fix it.

Akamai's not doing anything wrong.

OpenDNS is a broken concept.

Akamai only worked right 75% of the time even when it was first launched. Large ISPs that operated a single set of DNS servers for nationwide customers had the same problems.

All OpenDNS did was start offering the same thing ISPs already did as an external service.

How so? DNS was never designed to be used as a geolocation vector, as far as I have ever heard, so relying on it to be one does seem "wrong".

OpenDNS defaults to resolving non-existant domains as search results, bereaking DNS. You can turn it off, if you have a static IP.

I see. That doesn't seem to me to be a "broken concept", however, but a rather minor misfeature. I may have misread the poster to which I was responding, but his assertion seems to read to me that OpenDNS as a whole is fundamentally wrong or broken, and that Akamai is using the idea of DNS for something it was never intended to be is "correct". Both of those statements do not ring true.

This has nothing to do with Akamai being able to target users accurately. Google doesn't have this issue, so why does Akamai?

That's simply not true. Please argue with facts.

It's worth pointing out that this article is a bit long in the tooth. Although perhaps biased responses, it's worth looking at some other thoughts on the subject:

http://devcentral.f5.com/weblogs/macvittie/archive/2011/03/2...

http://dev.robertmao.com/2007/06/30/global-dns-load-balancin... (towards the end, and yes, the grammar is bad)

http://blogs.sun.com/davew/entry/thoughts_on_global_server_l...

The main reason seems to be clients caching DNS lookups for a period of time longer than what they should have, as indicated by the TTL field in the DNS response. If this behaviour changes across all browsers, wouldn't that solve this problem?

Another "hack" to solve the caching problem would be to have multiple random lookup records (say server-$random.hostname.com) all result in multiple lookups that cannot be cached? The tradeoff here is latency vs availability.

As mentioned in the article, triangulation and backup redirection would work as long as "Site A" can be up serving requests labeled (1).

What about using Anycast IPs? http://en.wikipedia.org/wiki/Anycast

Routing changes break the network connection. If it's buffered and you can reconnect transparently, it might work, but for unbuffered connections (like most web connections are) you'll see increased error rates.

This has been proven to not happen often in the real world.

http://www.nanog.org/meetings/nanog37/presentations/matt.lev...

My experience indicates otherwise.

Who are you? We thought we knew most folks who run TCP Anycast... Anyways, I've run multi-day long TCP streams without issue. Like any sort of anycast, there is a lot of finesse, but if you are logical in your network architecture you will have no issues. In fact, you can re-establish TCP connections across datacenters if desired using tools like pfsync.

I think one solution to this is to add a DNS record that says how to set the connection timeout for a given port for this IP before trying the next one. As is, there is no control over that and each browser implements it differently.

SRV records could do this, but most browsers can't do DNS. Only Chrome can.

What do you mean by this? Chrome doesnt support SRV for http http://code.google.com./p/chromium/issues/detail?id=22423 but all browsers do dns... No mainstream ones support srv for http.

Chrome resolves its own DNS. Therefore it can support any DNS record type (even if it doesn't have hooks for one today).

No other browser does. All other browsers rely on the system's stub resolver which means you can't even open a bug for them, because the bug would require you to speak DNS first.

Since the Amazon outage, I've been wondering about the best way to quickly switch to another host. Having multiple A records is so obvious I can't believe I have neither thought about it nor heard of it before (I've seen it implemented, but never thought about as a failover). I didn't know the browser would switch to the next IP. Interesting!

That said, can you control which host the browser will try first? I.e. can I have 1.1.1.1 as my main host and trust that all clients will connect to that as long as it's up, and only connecting to my backup 2.2.2.2 when 1.1.1.1 doesn't respond?

I have been wondering about the same thing. Sadly, the article says:

>but returning multiple A records diminishes any possibility of deterministic site selection.

Thanks mrcalzone, somehow I missed that piece of info. Too bad, that would have been super elegant.

I tried it, you can't control it, because some client's dns randomize such records.

It should work though if you use elastic IPs across different AZs in Amazon, or if you use geo ip and always send one IP for a different region that is close. Not clear if it is the best solution for Amazon though.

Thanks Justin - ideally, I'd like a setup with colocated servers and a cheaper virtualized backup at another provider. I've studied Learnboost's article about leaning on MongoDB's replication, I might reach out to them since I have a similar setup in mind, albeit on with a Redis backend.

Since the Amazon outage, I've been wondering about the best way to quickly switch to another host.

This is exact problem we (Cedexis) solve.

Assume your site is www.website.com, in general points to an A record or a CNAME to a datacenter/ cloud/ CDN.

We add an intermediate hostname, low TTL (20 seconds), global anycast network, which can be scripted (write your load-balancing logic in php) to handout a CNAME (one of many) based on performance (RTT), load, cost, anything else you can think of, etc.

Re.: AWS's recent outage, assuming you are running your apps in multiple zones/ regions/ clouds, we would have noticed the latency and automatically routed away to a different zone/ region/ cloud.

We collect hundreds of millions of performance measurements daily: http://gigaom.com/cloud/heres-what-amazon-outage-looked-like...

Drop an email and I am happy to explain more and setup folks from HN with a free account: prakash at cedexis.com

Thanks Prakash, I'll definitely check your offering and reach out when we have the need! It sounds cool even though I can't help but feel I would like really tight control over that part - basically we would be dependant on your service instead.

Reply on news.ycombinator.com