Hacker News Reader

Comment by animesh371g | Hacker News Reader

As someone mentioned, internally they use high performance load balancer like nginx, which provides most of the things such as TLS termination out of the box. How the machines are managed and scaled is dependent on the CDN provider. For eg:- Akamai would be using an altogether different approach than the one employed by Azure CDN.

View on HN

Nginx handles TLS termination, but OP brought up a good point - how do you distribute private keys to thousands of nginx nodes without compromising them?

Central store (something like Vault) with access keys for accessing the certificates/keys, small cache on the instances themselves (maybe few hours), monitoring with alerts about how often the central store is accessed.

But there must be something before ≈ nginx as well right? Something something BGP to route it to the closest POP? And when you have traffic to a single pop that are multiples of what even the beefiest machine can handle, something before the machine must load balance it I guess?

That’s done by DNS. The DNS server examines your DNS query and sends you back an IP address answer that’s “close” to the requesting IP address.

For example, if you are in country X, then even though there are dozens or hundreds of POP addresses for the name you requested, you’ll be answered with one for country X.

This can also be done with BGP. Anycast BGP for example is way better suited for this use-case.

BGP is an alternative to DNS but it is not better suited, it depends.

DNS gives more control and precision to CDN provider than BGP. BGP might be good enough if you got few POP around the world but with the scale/distribution of Akamai, DNS is better suited.

I think all major CDNs use some form of anycasting as it's pretty essential that you own your IP space. DNS-based load balancing can also be finicky as you'll have to deal with recursive resolvers, so a combination of anycasting and DNS-based load balancing probably works best.

How does DNS give you more control than BGP?

One is an actual routing system that tells routers where to send data, the other is a name translation system with multiple layers of caches outside of your control.

DNS is a layer above as BGP is still used to actually navigate to the listed IP, and any large CDN will own that IP space and announce their own routes anyway.

> DNS is a layer above as BGP is still used to actually navigate to the listed IP, and any large CDN will own that IP space and announce their own routes anyway.

Yeah, I'm not sure what parent is on about, BGP and DNS are not alternatives to each other, the internet relies on both of them but at different layers. Without BGP packets wouldn't know how to be routed and without DNS they wouldn't know where to be routing to, they are complementary.

Sure, Internet communications relies on both and you cannot swap one with another. This thread is about load balancing and both can be used for load balancing (Basically DNS resolving to Anycast IP vs DNS resolving to Unicast IPs).

What's the difference between an anycast IP and a unicast IP?

Answer: Nothing. They are just IPs.

Using a combination of Anycast and DNS is going to give the best control over steering http traffic. Particularly if you own a few prefixes and can do clever addressing tricks.

With BGP (especially anycast) you don't have direct influence where the requests lands. You can steer traffic by techniques like AS prepend, some priorities per BGP session (I'm not an expert in BGP) but eventually it's not CDN that decides where the request will be routed. It's decided by routers of the client's ISP and backbone networks, each taking decision themself (shorted AS path from his point of view + BGP policies set by network operators), not by you. You can't for example split traffic between DCs with specific proportion (like 30% here and 70% there). You can't split by something else than network properties like forward request to the region that has better chance to have this content in the cache.

With DNS and dynamic responses you are directing request to specific DC, even server, almost on every request. It may be dedicated for this traffic type (live stream different than static images etc). Your DNS server can take the hostname ("www.google.com") into consideration - BGP doesn't even know the hostname in the URL. If you wanted to do it with BGP you would need to place specific content to a dedicated /24 subnet and that is impossible considering how many IPv4 addresses are available.

BGP doesn't even consider network latency, current network load. CDN knows load on their machines, on their network link, where given content is placed. The bottleneck may be storage, network or CPU processing, different for different sites and content type. They need to direct traffic on request basis considering this and at least the hostname from the URL. That's why DNS is used first.

Anycast doesn't let you adjust traffic flows gradually to shed load: you end up breaking connections. DNS is more flexible but isn't always honored.

> That’s done by DNS. The DNS server examines your DNS query and sends you back an IP address answer that’s “close” to the requesting IP address.

Is this really true? Last time I was dealing with load-balancing and DNS queries, DNS was simply "round-robin" the replies, giving you back a random record basically of the ones replied. So if you have three A records with different IPs, each query will give you back one of them, but not depending on the location.

Maybe things have changed since I last dealt with it, but the DNS ecosystem doesn't tend to move very fast so I'm doubtful...

The DNS server doesn't have to return the same responses to every query. It can geolocate the address making the request and use that to determine the respone.

https://easydns.com/features/geo-dns/

(Having said that, I tried my toy site on cloudflare free tier from the UK and it gave me San Francisco IPs, so presumably they only do this for large enough customers)

> It can geolocate the address making the request and use that to determine the respone

It can, but it tends to be a premium feature of specific DNS providers, not a global/by-default feature of DNS as efitz seems to be alluding to.

DNSimple supports it for example, but only on their "Professional" plan (and they call it "Regional Records") while others like Gandi don't support it at all.

If CDN providers are dealing with BGP you'd think they would also run their own DNS servers. I think you can do it with BIND views, from memory.

I think that all of Cloudflare's IPs are GeoIP'd to their main office, like how Google's are. You can see the POP you're hitting here: https://<your url>/cdn-cgi/trace under colo. I think that Cloudflare use Anycast instead of geodns though.

BGP Anycast (possibly in combination with DNS depending on geolocation).

Basically you announce the same IP address from multiple locations, and BGP chooses the best* Route to reach that address.

* Note that "best" does not nessecarily mean "fastest", not even "least hops" - BGP has many knobs where the routing decisions can be manipulated, but by default it choses the route where traffic has to traverse the smallest number of "Autonomous Systems" (roughly equivalent to "Organizational Entities").

Reply on news.ycombinator.com