Worldwide DNS Performance
dnsperf.com
dnsperf.com
1. The point of the service is to have an objective way to compare different DNS services. dnsperf is supposed only to help you in your search for the best DNS service and not crown the absolute winner.
2. I plan to add as many locations as possible to get more reliable data. More locations will also solve the problem of "DNS at the same datacenter as my test nodes"
3. Some people are confused about the node locations. Here is the map http://www.dnsperf.com/network
4. Its not open source but some time in the future it will be.
5. The tests I am running are supposed to exclude variables such as local/ISP caching, DNS proxies, resolvers, SRTT and so on. I wanted to test the nameservers directly and get the raw performance data of the provider himself. I believe thats the best way to do fair comparison.
Thank you for your comments and love. I'm glad people liked the service :)
Feel free to email me with your feedback or questions.
Thanks for putting this service together. For an African testing location maybe a VPS provider from ZA would work? Following are a few providers with Joburg or Cape Town presences (haven't personally used any though so YMMV):
http://www.vpsnine.com/hosting/locations.php http://www.webafrica.co.za/hosting/vps/ http://www.webnow.co.za/vps/ http://www.web-telecoms.co.za/vps/
(posted these to the comments on your site but re-posting here in case you miss it).
Also, and I'm hoping someone here can clear this up for me, but it's not clear to me how these results translate to actual DNS performance for my site. I only vaguely understand how the magic of the internet works but don't the DNS results get distributed throughout the network making the performance of the root servers less relevant? If I could chop 50-100ms off my TTFB by switching my DNS from Namecheap to someone else, I would definitely pay for that, but it's not clear to me.
Admittedly, not much of an issue if you're a big site with hundreds/thousands of requests a minute but it is an issue if you're a less trafficed site and have a low TTL.
By simply adding more locations I fix the problem, add more data and simplify the whole process.
What the graphs actually display is latency - in which case less is better.
That was confusing for a few moments.
Hopefully this data is trended over daily, weekly, monthly, quarterly, and yearly rollups. I see monthly when drilling into a provider, but that's all. A rolling annual graph would really tell a true story of how consistent a provider is and that, in itself, is very important when considering outsourcing DNS.
It'll be more realistic then DC -> DC queries.
- http://www.w3.org/TR/resource-timing/
- http://googledevelopers.blogspot.se/2013/12/measuring-networ...
That depends on what you're trying to measure. If you want to know what the total end-user experience is like, Resource Timing is awesome.
If you're trying to compare DNS providers without noise from the differences in client implementations and ISP competency, however, it'll be very noisy and you need a massive number of samples to avoid seeing trends which are really just sampling artifacts.
The best approach would be to do both so you could compare the server response times with the measured client values so you can get an idea whether a slow-down this week is caused by something you have any control over.
This should really be taken with a grain of salt as Asia is a big place, and unlike Europe or the US, the interconnects between countries are poor. From where I am in the Middle East it's faster to ping Europe (~180ms) than Central Asia (~250ms). I'd love to see a report that takes this and CDN performance into account on a country-by-country basis!
You can join, host a probe (get it sent to you, plug it in) and then run your own measurement tests based on the 'credits' you earn from having the probe running.
There are about 400m people in south america, and this test declares a provider to be the fastest for South America because both the provider and the test taker happened to pick Sao Paulo.
Speed is is the magnitude of velocity so bigger is faster.
What is displayed here is the time taken - the reciprocal of speed.
Personally I run dnrd, a dns caching proxy http://dnrd.sourceforge.net/
If you're going to be pedantic, velocity is a vector, and vectors must reside in a vector space, so what vector space do the velocities in this context reside in?
Ignoring all context to call into doubt the use of a perfectly well-understood word adds nothing to the discussion.
It's always neat to see nice data collection like this, but unfortunately the average speed to the authoritative name servers isn't a very meaningful measurement. Real world resolvers bias heavily towards the fastest name server for your zone, and they are so latency sensitive that they'll do things like issue concurrent queries to several name servers at the same time.
The upshot of that is that what really matters is the latency to the closest name server, or at worst the latency to the 3rd fastest server; for the rare boot-strapping cases. Bind, the most common resolver by far will issue up to 3 concurrent queries to different name servers as part of its SRTT algorithm. The next most common resolvers; Unbound, OpenDNS, and Google Public DNS perform pre-fetching and so the latencies aren't contributing to the user experience except for extreme outlier queries.
Some large DNS providers design to this behaviour, and seek to increase the average distance to their DNS servers by operating the name servers for each domain in different data centers. That gives routing and path diversity for the DNS queries and responses. Since network path diversity increases with distance, this works best when you include a location or two that are quite far away, which increases the average latency to those servers - but thanks to resolver behavior doesn't do much to the user-experience.
A write up for Route 53's consideration of the trade-offs is here: http://www.awsarchitectureblog.com/2014/05/a-case-study-in-g... (there's also a video about the role this plays in withstanding DDOS attacks: https://www.youtube.com/watch?v=V7vTPlV8P3U around the 10 minute mark).
Where the average latencies are low, all of the name servers are in close proximity to the measurement point and I would wager that the network path diversity is probably quite low. A small number of link failures or ddos/congestion events, maybe even one, might make all of the servers unreachable.
A more meaningful measurement of the speed itself is to perform regular DNS resolutions, using real-world DNS resolvers spread out across your users. In-browser tests like Google analytics go a long way here, and it's fairly easy to A/B test different providers. The differences tend to be very small. Caching dominates, as others here have mentioned.
Apologies if I seemed to rain on dnsperf's parade here; it's a neat visualization and measuring this stuff is tough. It's always good to see someone take an interest in measuring DNS!
The RTT mechanisms in resolvers have a high degree of randomness and will aggressively try the other, slower name servers again. E.g., out of 1000 samples, my desktop in the Netherlands (via XS4All) sees low latencies from Route 53 ~60% of the time:
$ seq 1 1000 | xargs -n 1 sh -c 'dig test-$0.trosc.com | grep "Query time" | awk "{print \$4}"' | histogram.py -f "%3d"
18 - 59 [ 609]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
59 - 100 [ 1]:
100 - 141 [ 1]:
141 - 182 [ 229]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
182 - 224 [ 102]: ∎∎∎∎∎∎∎∎∎∎∎∎
224 - 265 [ 37]: ∎∎∎∎
265 - 306 [ 1]:
306 - 347 [ 15]: ∎
347 - 388 [ 2]:
388 - 430 [ 2]:
As you pointed out, Route 53 optimizes for availability and DDoS resilience over RTT performance. There are 4 name server IPs to choose from which gives me 4 different paths to 4 different server locations via anycast, giving me 4 different RTT buckets. Few DNS providers go through such lengths for availability. Still, 185ms is a lot. It's probably because anycast/BGP advertisements from the US get to AMS-IX with a smaller number of hops than competing advertisements from European locations. I would guess Route 53's current striping is not heavily tuned for RTTs.
Caching solves part of this, but there are a lot of resolvers out there. As a thought example: Assume your sources of traffic are uniformly distributed among 75000 resolvers and you use 60 second TTLs (pretty standard), then you won't see significant benefit from caching until you get to >>1000 requests/s.
Many applications also have a long-tail of DNS names and basically wont benefit from caching at all. This could be motivated by availability as well (think shuffle sharding :). I'm building one where DNS query time currently dominates page load time (especially aliasing to CloudFront can be slow :). It's useful to understand that there's a general availability vs. latency trade-off in DNS that only gets partially addressed by the resolver.
I think your argument is flawed because your users are not uniformly distributed among those 75000 resolvers.
In practice just ~1% of the resolvers (Comcast, NTT, Telekom, etc.) are handling >90% of your users. Consequently the benefits of caching kick in much earlier and stronger than you suggest.
The 75000 was mostly a thought example, it's very hard to know what a good number is, although there is a Route 53-related reason for that number. In any case, the benefit of DNS caching is probably much less than you think due to short TTLs and the number of resolvers.
I assume all major ISPs use 2 or 3 layers of cache, which makes the size of their perimeter fleet largely irrelevant.
it's very hard to know what a good number is
Could you perhaps ask your former Route53 colleagues for some log-file insight?
the benefit of DNS caching is probably much less than you think due to short TTLs and the number of resolvers.
I don't think so. The overwhelming majority of clients uses their ISPs resolver. So all it takes is one hit per major ISP per TTL to keep it zippy for almost everyone. That's why DNS works so well, after all?
Not really. The resolvers tend to be geographically dispersed and use anycast. Having a multi-layered cache would probably decrease performance, except within a specific location.
There are some nice research papers studying DNS resolvers, e.g. here's one for cellular networks: http://www.aqualab.cs.northwestern.edu/component/attachments...
Could you perhaps ask your former Route53 colleagues for some log-file insight?
They see what's behind the cache, not how much traffic the resolvers are taking. Could be the same, could be 100x more, hard to tell.
So all it takes is one hit per major ISP per TTL to keep it zippy for almost everyone. That's why DNS works so well, after all?
Caching works great with long TTLs, e.g. as used for NS, MX, CNAME records. The problem is the 60 second TTLs that are commonly used for A records in cloud services. Except for reasonably high volume names, it's not highly probable that your A records will be in a given cache at a given time. Many applications also use many different domain names (e.g., one per user), which creates a long tail of low volume names.
Of course, traffic is not uniformly distributed in any way, so there might be parts of the day when your name will be constantly served from cache everywhere, or parts of the world where it is never served from cache.
That said, there are a number of problems with this:
1. As stated by other people here, this is only from the perspective of where the tests originated, and is somewhat irrelevant once the DNS result is cached by your ISP.
2. If you are using any of a number of modern browsers, DNS will be prefetched before you even click on any link, so the DNS time from a user perspective will be zero the majority of the time.
3. The point of this was originally for choosing a good host for shared JS files. It is much more meaningful to use a distributed host with location based DNS to point you to the nearest fastest server, rather than just making the DNS request itself as fast as possible. Why do you think Google has so many servers all around the world?
We're thrilled to see a new DNS performance monitoring tool in the public domain. It takes courage to release things like this to the world and open them up for critique, I applaud jimaek, the author, for doing just that.
As many have noted, the results are highly varied from many of the other reports we see today but more locations and tests are going to continue to improve accuracy.
The testing is synthetic, that is, it's run from within datacenters. I'd encourage anyone who can help provide/sponsor a VPS or resources to also do so. Providing consistent benchmarks against Authoritive Queries from know locations is great but the test network itself needs to expand to get truly meaningful comparisons between providers.
I'd hope to see real-user data also get added in the future, that is, testing on the end of home cable/broadband connections, these tests perform differently to synthetic tests but represent end-user experience more accurately.
It would also be great to see raw lookup performance between providers, exclusive of network latency.
For those interested in diving deeper into DNS performance, I'd also recommend reviewing Cloud Harmony's reports, the latest of which can be found here: https://cloudharmony.com/reports/state-of-the-cloud-dns-0315...
DNS latency is only one factor in overall perceived performance, and it can be affected greatly by routing, traffic load, and other factors. Additionally, many requests you'll make will likely go through a resolver with caching anyhow, so that too can have a significant impact on perceived latency.
Thus DO (or any provider for that matter) only really needs to provide reasonable resolution times from their servers to be effective.
A more important matter is consistency, i.e. are they consistently returning responses or are they actually dropping packets due to load. With a service that's growing quickly, keeping up with the increase in load on their network probably presents a more significant problem.
I'm not sure why they're still so high though.
For this kind of measure, it would really be better to use either a median or, better, reporting multiple percentile performance. In my experience, DNS timings have extremes which might be surprising (e.g. most people see 50ms but 10% of visitors see 5+ seconds) and a few large samples will distort an average.
Your 2 second cap will cap the distortion, which is much better than e.g. Google Analytics which will merrily record multi-hour timings, but I'd really prefer to see e.g. 50th, 75th and 95th percentiles to see whether a particular provider might have better worst-case performance than another.
I've been very happy with DNSMadeEasy (although I think the name somehow diminishes it's technical viability somehow - not sure why, but it sounds "cheesy" which is not reflective of the service at all). I am very glad to see them in the top 10 for the most part, they offer a fantastic product, well documented easy to use API and great management panel.
For comparison I've tried Zerigo, PowerDNS, UltraDNS, Dyn and Route53 and dropped them all for either cost or technical reasons. For my small use cases I've found DNSMadeEasy to be a nice middle-ground.
Would be interesting if the OP could add some sort of subjective rating options from outside sources in terms of user reviews. Could rank the providers on performance (latency), cost, functionality (API, Anycast, etc) and management.
Dyn's early results were not representative of our network or service. This was as a result of the limited number of nodes currently testing and the location of these nodes. We found a routing issue between Azure and ourselves via one carrier leading to elevated results for a short period of time.
There's been great commentary on here about providers showing as fast because of the proximity of the tests. This will only be rectified by adding more monitoring and a broader base of test locations.
I know jimaek is working on getting new locations online - if anyone here can assist in providing or sponsoring test locations, I encourage you to reach out to him.