Comment by eat_veggies | Hacker News Reader

Comment by eat_veggies | Hacker News Reader

Parent Full thread

One of my favorite revelations about the network tracing tools (things like `traceroute` and `dig +trace`) that might not be obvious for people like me who work higher up in the stack, is that the data they provide isn't usually made available during "normal" usage. Packets don't just phone home and tell you where they've been. Something else is going on.

When you send a DNS query to a recursive server like your ISP's or something like 1.1.1.1, you make a single DNS query and get back a single response, because the recursive DNS server handles all the different queries that Julia outlines in the post. As the client, we have no idea what steps just happened in the background.

But when you run `dig +trace`, dig is actually pretending to be a recursive name server, and making all those queries itself instead of letting the real recursive name servers do their work. It's a fun hack but that means it's not always 100% accurate to what's going on in the real world [0]

[0] https://serverfault.com/questions/482913/is-dig-trace-always...

Traceroute’s trick is amusing. It abuses the TTL field, sending out packets with too-low TTLs and waiting to see who complains about them. When layers reveal themselves they are doing it voluntarily, and those wise to the game can choose not to participate, or to troll it.

https://www.theregister.com/2013/02/15/star_wars_traceroute/

Those are IP packet TTLs. Also useful in some cases for diagnosing mysterious RSTs and SYNs. Also useful in some cases to purposefully set low if there is route flapping.

The IP TTL field is unrelated to the topic - DNS and IP are sitting on the different OSI layers. Also, the traceroute command does not abuse this field, it’s just the way the IP protocol works. I.e. eBGP sets TTL to 1 by default, to prevent establishing relationships with the neighbors further that one hop away.

It's not abusing the TTL, it's working as designed

AFAIK the design purpose of TTL is to prevent an infinite loop in case the route contains a cycle.

Just to add to the discussion -- 'whats happening in the background' -- more specifically is your operating system's stub resolver.

So when you ask for www.amazon.com it ends up making multiple DNS lookups, as www.amazon.com is a CNAME record.

Nothing about this CNAME lookup gets passed back up the stack to your application; you just get that end-result: the IP address.

host www.amazon.com www.amazon.com is an alias for tp.47cf2c8c9-frontier.amazon.com. tp.47cf2c8c9-frontier.amazon.com is an alias for www.amazon.com.edgekey.net. www.amazon.com.edgekey.net is an alias for e15316.e22.akamaiedge.net. e15316.e22.akamaiedge.net has address 23.204.68.114

Am I the only one who finds it strange that Amazon uses Akamai instead of Cloudfront?

It varies. For example, for www.amazon.com, I get www.amazon.com -> tp.47cf2c8c9-frontier.amazon.com. -> d3ag4hukkh62yn.cloudfront.net.

Why is that?

(My first thought was, wow, surprised they outsource that.)

I'd wager at least part of it is the same reason many companies use CloudFront: Separate point of failure in case of downtime at the AWS side, some one else shouldering DDoS.

Pure speculation, but probably the same reason they ran Oracle for so long -- not many cdn providers when amazon.com launched. Afaik akamai still has one of the biggest edge networks of all the providers.

I asked the same question 7 days ago: https://news.ycombinator.com/item?id=23537135

Yup, and to complicate matters more those resolvers you're talking to may be talking to more caching resolvers.

For a given application server it might be:

- check local dns caching resolver

- check local network caching resolver

- if internal domain, check local authoritative resolver

- if public domain check isp resolver

- recursively resolve from there

In particular, one your ISPs/their ISP's DNS servers may be caching a record for longer than it's supposed to and will return incorrect and expired data.

The other possibility is different IP's being returned by a DNS server based on where a query is coming from, eg a CDN. If you're in location A and your ISPs DNS server is in location B, the CDN's DNS server may return a different IP based on if the request is coming from A or B. ECS [0] is supposed to mitigate this, but may or may not be used.

[0] https://en.wikipedia.org/wiki/EDNS_Client_Subnet

> In particular, one your ISPs/their ISP's DNS servers may be caching a record for longer than it's supposed to and will return incorrect and expired data.

It's disturbing how many clients we'll see hitting an old IP address for 30 days after a change.

My favorite dig trick is asking caching resolvers with +norecurse to see what's in the cache. Of course some resolvers don't honor it, just like they don't support ANY.

I thought DNS was a UDP protocol where you fire off a packet and use the first response that comes back. It's not always one to one.

UDP is stateless but DNS is definitely 1:1 in that there is a notion of a query and its reply. This relationship is made even clearer when you consider DoH.

dig +trace takes one path, there’s also tools like dnstrace that attempt to show all the paths: https://github.com/rs/dnstrace

Still there can be caches that don’t quite agree as the other comment mentions.

Reply on news.ycombinator.com