Hacker News Reader

> Aside: asking how traceroute works is one of my interview questions, most people don't know...

Curious what type of roles you interview for, are they networking-centric? Iirc this is CCNA-level material, I'd expect anyone working in networking to be able to describe how traceroute works. I've used it more as a smoke test question than a question that most people don't know.

Unix sys admin, build & release, developer tooling, operations.

We use the Internet every day. I like folks to have an idea how it works, to be intellectually curious, and to be generally informed about the technology they use.

The traceroute man page explains how it works.

FWIW, my CS degree included a networking class.

fwiw, networking wasn't a required class at my school. neither was databases, weirdly, nor anything web-adjacent (no JS, no HTML or CSS.)

...huh. I'm realizing my CS department was pretty weak. but tbf I knew how traceroute worked by high school, so it didn't matter.

Describing how it works and using it to troubleshoot real problems and drawing the correct conclusions are very different things. If you believe it is simple then you have a shallow understanding of traceroute, which is why it's a good interview question as you can go really deep on your answer.

Even seasoned network engineers get it wrong.

https://youtu.be/WL0ZTcfSvB4?si=dtihJG1pGJwiyj1u

Your linked 1+ hour video includes things like IATA airport codes for geolocation and such. None of that is relevant to the fact that all traceroute implementations make use of very basic networking fundamentals: in particular, by sending out a series of packets with artificially limited TTL, to [hopefully] receive ICMP type 11 code 0 "TTL expired" packets from the hops along the path to your target.

Traceroute is simple. Sure, there's lots you can do to enrich the data you receive (e.g, reverse DNS and geolocation), or sending multiple sequences to identify equal cost multipath. But these are not inherent or necessary to perform a traceroute.

And understanding why different protocols exhibit different behavior / observe different metrics, or why some nodes don't send ICMP TTL expired, is important. But that's more in line with what you call "using it to troubleshoot", which is not "how it works."

But "how traceroute works" is simple: First you send a packet with TTL=1, then you send a packet with TTL=2, and so on. That's it, that's how it works.

The typical candidate I might ask doesn't know that IP packets include a TTL field. So we start simpler: "What is a packet?" "What is a router?" "How does a packet get to the destination?" "What if there's a routing loop?"

Some candidates throw up their hands immediately, which makes it a short interview. Some candidates already know, which makes it a useless question and we move on to other things. For everyone else, I think it's a good interview question.

Yup I saw your other reply and totally makes sense -- networking related or adjacent, but not quite networking-proper type roles. Thanks for your replies.

I asked chatgpt to give me answers to these questions as I'm also in the process of learning these things. What do you think of these answers? Would chatgpt pass the interview?

1. *"What is a packet?"* - A packet is a data unit sent over a network, encapsulated within protocols like IP and TCP/UDP. It contains both the payload (actual data) and control information such as source and destination IP addresses. Packets enable efficient routing and reassembly at the destination in network communication.

2. *"What is a router?"* - A router is a device that forwards data packets between networks, operating at the network layer. It uses IP addresses, routing tables, and algorithms to determine the best path for packet forwarding, connecting different network types and managing traffic between internal and external networks.

3. *"How does a packet get to the destination?"* - A packet reaches its destination through routing. It passes through routers that consult their routing tables to forward the packet. The packet traverses multiple networks, with IP protocols guiding it towards the destination, ensuring error checking and congestion handling.

4. *"What if there's a routing loop?"* - In a routing loop, a packet is passed continuously between routers. The Time-To-Live (TTL) field in IP packets prevents endless loops by decrementing each time a packet passes a router. If TTL hits zero before the destination is reached, the packet is discarded to prevent network clogging. Routing protocols also have mechanisms to detect and prevent loops.

Its relevant to what I stated, that interpreting the data that traceroute provides is not simple. The mechanism itself is indeed very simple.

Have you never been asked a question in an interview that starts a discussion or has follow up questions? In isolation its not a good question, true for most questions, but to initiate something deeper its good. After the initial explanation of how it works you can get into how you have used it, what kind of issues you have solved with it. Then maybe look at an actual case and give your interpretation of the data. You could get into router hardware architectures, what the control/data planes are, why some drops in the output are not a problem and when they are, ECMP, why bidirectional traceroutes are useful, routing topology, flapping routes, etc.

I just realized I don't know traceroute as well as I thought I did. apparently Unix systems send UDP packets, while Windows sends ICMP. which maybe explains why I've had such a miserable experience with traceroute on the modern Internet, because many ISPs block ICMP from outside.

I’ve never encountered an isp blocking icmp that doesn’t also block all sorts of traffic. I can ping 8.8.8.8 or whatever from almost every connection I’ve used anywhere in the world.

Not every router will return a tel expired of course, and many ISPs route ipv4 traffic via rfc1918 addresses nowadays , so you can get big gaps in their networks, but that applies whether your outbound packet is icmp, tcp, UDP, or any other type.

Worth remembering that packets with different source and destination ports can route via different paths, so sometimes you need to be aware of the entire ip/port/protocol for src/dst and configuring them properly. Nat can cause problems there too when it changes your source ports.

+1 for this, I always recommend ras' tutorials at NANOG. He goes into a lot of detail people just hand wave over or don't even consider

Hadn't come across NANOG before -- thanks for mentioning them.

Here's a link to their videos: https://www.youtube.com/playlist?list=PLO8DR5ZGla8iVN2v3UKkR...

This is the ras' "Troubleshooting with Traceroute" tutorial: https://www.youtube.com/watch?v=WL0ZTcfSvB4&list=PLO8DR5ZGla...

Slides: https://archive.nanog.org/meetings/nanog47/presentations/Sun...

(There are one or two other traceroute tutorials, not sure how different they are from the above, e.g. https://www.youtube.com/watch?v=4dUqVlZ6trU&list=PLO8DR5ZGla... ).

> Also, Linux/Unix traceroute by default use UDP to a high numbered (and usually closed) port for probe packets instead since UDP historically is less likely to be dropped/filtered than ICMP.

thanks! ive been wondering about this for ages!

Trivia: 32,768 + 666 is the usual starting port of most implementations

Is that still true? https://www.rfc-editor.org/rfc/rfc3443.txt has no mention of seconds.

It just says that outputTTL is (inputTTL - 1). With some exceptions.

[edit: I missed that that RFC is for MPLS but would be interested in your comment anyway; the definitive version seems to be https://datatracker.ietf.org/doc/html/rfc1122]

RFC 791:

> The time is measured in units of seconds, but since every module that processes a datagram must decrease the TTL by at least one even if it process the datagram in less than a second, the TTL must be thought of only as an upper bound on the time a datagram may exist.

https://www.rfc-editor.org/rfc/rfc791.html

The equivalent field in IPv6 is named hop limit in recognition of how the TTL field is used in practice with IPv4:

https://www.rfc-editor.org/rfc/rfc2460.html

BGP looking glasses are very fun, although a bit of a different concept.

And WOAH, that's really interesting about TTLs. Thanks for sharing! That's awesome and terrifying!

The looking glass sites often include a traceroute tool too, but "looking glass traceroute" may return better results. Unfortunately most of them have bitrotted. :-(

> many are unable to figure it out from first principles no matter how many questions I answer about TCP/IP

TBF it's neither TCP nor IP, but ICMP :-)

Edit: I meant to write "neither TCP nor UDP" -- but even though I could make the correction I'll leave my error in place.

It's IP and ICMP and possibly UDP, which are all part of TCP/IP, aka the Internet protocol suite.

The traceroute that ships with macOS supports ICMP, UDP, TCP, and GRE. As I recall, the one(s) available on most Linux distributions do as well.

> ipv4 TTL is dejure seconds

Huh. TIL. But how would a router know how long a packet took to traverse the hop? The packet doesn't have the information to figure that out … were they expecting people to configure routers to know how far away the previous hop was?

Also >1 second is most of the way to moon. (Yes, yes, speed in a non-vacuum is blah blah and switches blah buffers blah…)

Here's what Jon Postel wrote in IEN 41[1] in June 1978:

> The time to live is set by the sender to the maximum time the segment is allowed to be in the internetwork system. If the segment is in the internetwork system longer than that the segment should be destroyed. This field should be decreased at each point that the internet header [is] processed to reflect the time spent processing the segment. Even if no local information is available on the time actually spent, the field should be decremented. The time is measured in units of seconds (i.e. the value 1 means one second). Thus the maximum time to live is 255 seconds or 4.25 minutes.

A router would have to track of when each packet enters and leaves and then round to the nearest number of seconds.

[1]: https://www.rfc-editor.org/ien/ien41.pdf