Passive observations of a large DNS service | Hacker News Reader

Passive observations of a large DNS service

blog.apnic.net

56 pointsxuande21 comments

"Due to the centralization that is caused by public DNS services, large content delivery networks (CDNs), such as Akamai, are no longer able to rely on the source IP of DNS queries to pinpoint their customers. Therefore, they are also no longer able to provide geobased redirection appropriate for that IP."

I always thought that the go-to way to solve geobased redirection was to make all your PoPs listen to the same IP(s) (I think some BGP is involved here), and let Dijkstra's algorithm let all clients find the shortest path to the nearest PoP? Isn't this exactly what Google and CloudFlare are doing with their own DNS IPs?

That technique is called "Anycast", it's mentioned at the very end of the linked article.

The problem with Anycast is it works pretty good for UDP (such as DNS) but doing Anycast with your actual TCP connection (particularly long lived ones) can be problematic because of shifting routes, multipath, etc.. meaning that sometimes your traffic will change data-center mid-flow or possibly even with alternating packets in the case of multipath somewhere etc. So suddenly your HTTP connection data is arriving at the wrong data center and the connection breaks.

What most providers currently do (as far as I understand anyway, I'm sure there is a lot of hyper optimization) is based around using Anycast to receive the DNS lookup, and use information from that DNS lookup to send back a relevant POP locations IP.

That's not always based purely on which DC received the DNS request though, that's the most naive implementation, but most of the big CDNs optimise the returned server based on the source IP of the lookup among many other factors.

Hence why it matters for public DNS, since they are the source of the DNS request.

This article however also discussed that Anycast doesn't always route traffic where you would "expect", for various reasons that extend from commercial agreements and cost all the way to bad configuration and "no idea" you may not get routed to the nearest DC.

There is also a DNS extension, called "EDNS0" (also discussed in the article) that sends the /24 of the IPv4 IP along with the query, to allow the DNS resolver to customize their response particularly for these centralized DNS services (e.g. Google DNS supports this but only if you apply to receive them as an authoritative DNS server) - however APNIC/CloudFlare explicitly chose not to use this I believe supposedly for privacy reasons with 1.1.1.1.

Very informative answer, thanks!

The problem of running TCP over Anycast addressing tends to be overblown. Cloudflare, for example, does all of their client-facing HTTP traffic over Anycast and so you can consider that as existence proof for it being a viable strategy. Routes simply don't seem to shift all that frequently in practice.

I used to operate an anycast DNS service and we monitored it from probes all over the world. Sometimes a random ISP in USA/Europe would start sending queries to Japan and that could last for seconds to hours. I'm not sure how often it happens, but it will certainly be enough that it will cause broken connections for a few users.

I'm not sure how Cloudflare deals with it - potentially they could share the TCP connection data between datacenters when needed.

Or maybe it has something to do with their how edgy their setup is - if it's mostly intra-ISP cache servers being anycasted, the routing should be more stable.

AFAIK a lot of Google’s HTTPS traffic is also routed using anycast. They also offer this to GCP customers under GCLB (https://cloud.google.com/load-balancing/).

I don’t know how they deal with changes in routes.

Some interesting anycast stability measurements: http://tma.ifip.org/wordpress/wp-content/uploads/2017/06/tma...

Anycast still has enough problems, that it's a miracle some are using it more or less successfully. It's not like pure DNS-based routing, where you can throw a bunch of cheap dedicated or virtual servers at it in various countries, measure dns/http latencies in the browser and then just choose lowest latency nodes for IPs, subnets.

Thank you sir, very helpful explanation especially about EDNS0.

I am using DNSCrypt except queries for some resources that are CDN based like youtube, HBOGo cdnjs etc which were forwarded to 1.1.1.1. I have changed to google 8.8.8.8 now and made sure that they can see my ECS. Thanks

dig +short TXT whoami.ds.akahelp.net

Pedantic side note: EDNS0[0]is the name of the DNS extension mechanism, of which “client subnet”[1] is but one use of EDNS0.

0. https://tools.ietf.org/html/rfc6891

1. https://tools.ietf.org/html/rfc7871

Are out-of-country DNS queries necessarily bad? If I were in upstate New York, for example, I would much prefer to use a resolver in Toronto than one in San Francisco. Measuring actual distance or network latency would be a much better way to detect suboptimal geolocation than simply checking whether the server is in the same country as the user.

They could definitely hamper mechanisms that use your location to enforce DRM, like video streaming services.

I assume that most parties that enforce DRM will look at the source address of the TCP connection, not what instance is accessed.

Too many people know about DNS tricks to get the 'right' instance.

Additionally, most VPN providers are blocked even if they are geographically in the right place.

> Measuring actual distance or network latency would be a much better way to detect suboptimal geolocation

And how would you measure that in case of DNS servers? What you mean by distance ? Amount of hops? Physical distance? One hop can have the same latency as route with 3 hops. Physically closer POP can have higher latency than one that is further away. And we didn't even touch the throughput.

It's not so easy as it sounds.

> What you mean by distance ?

I believe the person you're responding to meant miles. Physical distance.

> It's not so easy as it sounds.

Their point was not that it isn't easy (and I think they realize that) but that the article seems to imply that out-of-country is suboptimal, and they're questioning that assumption. (Hence the SF vs. Toronto from upstate NY example.) DNS resolvers are already computing some form of location to determine whether a query is in or out of country, and presumably the resolver knows its own location, so it effectively is already solving (to a degree) the problem of computing distance; the poster's point was just that "in country" is not necessarily "closest", "fastest" or "best" (or even that those three are the same).

> DNS resolvers are already computing some form of location to determine whether a query is in or out of country

Most of it is based on publicly available IANA databases (RIPE, ARIN etc) which are not always accurate, as IPs are traded and moved between AS's all the time + dynamic IPs that are given to users from example both coasts of USA.

> the poster's point was just that "in country" is not necessarily "closest", "fastest" or "best" (or even that those three are the same).

And I wrote it's not easy to solve because you do not get accurate GPS positions to most of IP clasess, which means it's a lot easier to point to the same country and that's why almost every provider does that. Some use geo coordinates of IPs but only for certain classes that have accurate positioning and are inside the same country.

PS. I am building globally distributed anycast GEO-DNS service/system at the moment.

Welcome to Australia. (-:

* http://apcmag.com/why-using-google-dns-opendns-is-a-bad-idea...

I would naively assume that with the huge number of POPs that Google has and the desire to provide a low latency service, Google would only need to have the queries that originate from their resolvers have the right local addresses.

What might be a problem is that for those addresses there is no geolocation information. Which Google could solve by supplying that to maxmind, etc.

Hence the need for the privacy violating subnet option.

It already does. There's a list of exactly that, supplied by Google, hyperlinked-to in the very article headlined here.

I don't feel comfortable with DNS services being "observed".

Do something about it then.

Reply on news.ycombinator.com