The journey of an internet packet: Exploring networks with traceroute | Hacker News Reader

The journey of an internet packet: Exploring networks with traceroute

sebastianmarines.com

Traceroute doesn't see 90% of the machines your packet passes through.

When your packet leaves a router at some-pop-some-port-wherever, that fiber isn't usually the same piece of glass that plugs into the next hop. There's a whole chain of amplifiers and possibly multiplexers that handle it between here and there.

Some of those provide reliable transport service, giving you the illusion of a fiber that never breaks, despite backhoes doing what backhoes do. Some of those shift the wavelength of your signal, letting you use cheap optics without troubling in the nuances of DWDM that packs your signal alongside dozens of others onto the same long-haul fiber. Some of those just boost the signal, along with all those others on the same fiber.

But what all those machines have in common, is that none of them speak IP. None of them touch the payload. None of them are capable of decrementing a hop count. They're "part of the wire" as far as the packet is concerned.

In my experience, this leads to two types of network engineers, separated by their understanding of these underlying realities.

In traceroute’s defense, it is traceroute — for sure it doesn’t tell you anything about the devices that don’t operate at the IP level. Those devices either don’t affect the IP “route” abstraction (e.g., signal boosters) or do so in ways that end up plausibly visible in the next hop.

There’s a reason the network layer abstraction is so strong, and an analogy to CPU ISAs here that have a similar strength.

TCP, similarly, doesn’t tell you when packets are deduplicated/resent/reordered/etc. — that’s just not part of the presented abstraction. Want that? Use UDP.

"It doesn't show me the local digital loop carrier!"

"Is the digital loop carrier doing any IP-level routing?"

"No, but..."

> In my experience, this leads to two types of network engineers, separated by their understanding of these underlying realities.

What's wrong with that? Certainly, someone with a complete picture is "better", but it is effectively two different types of problems. Do they need to be combined?

I think you're reading the connotation into this. In my opinion, there is nothing wrong with being a higher level engineer. There's plenty of work "above the fold" so to speak.

For example, I think it behooves every software engineer to have a general grasp of how CPUs work, what speculative execution is, how CPU caching and invalidation works, etc, but the average webdev doesn't really need to know this, and might run into some abstraction breaking implications only a few times in their career, while debugging a tricky bug or performance regression.

I imagine something similar is true for network engineers. Likely many can work for years at a time without worrying about fiber signal repeaters, other than that one weird packet loss issue that ends up getting traced back to a marginal optic in a cable vault somewhere.

Of course, none of this applies to the compiler engineers or the people who build the physical network layer. They are in the "second type" of engineer that actually needs to understand this stuff in depth in order to do their jobs on a day to day basis.

> I think you're reading the connotation into this.

I am reading the connotation into this and asking about it. 4 paragraphs of talking only about tech and then it diverged into a personal statement at the end.

What I meant by that statement is that whatever you're asking about is actually coming from you. I don't think the simple statement in the comment actually contains any connotation. Just like the old programmer joke - there are 10 kinds of people in the world, those who understand binary and those who do not. We just like dividing people into classes because it helps us understand the world.

I guess to engage with this a bit more, one reason you might find a negative connotation towards engineers who are ignorant of the underlying details is that it is the larger set. After all, most engineers do not need to understand the physical layer in depth, so there are fewer who do. People love to feel like they're part of a smaller "higher" class, for complex social reasons. This sometimes comes with a bit of a distaste towards those who are part of the "regular" class.

But overall, I think you're taking this further than it really needs to be taken. The GP was just saying that not all network engineers are wizards in the technical details of lower layers, just like not all software engineers can write a compiler. Does that make them "worse" in some way? Well it makes them a worse fit for that few specific jobs that require that extra knowledge, but in general, I don't think so, and I doubt most people do either.

That's why we have abstractions.

When everything's happy and packets are flowing as they should, there's absolutely nothing wrong with living at whatever layer of abstraction you wish.

But when something is broken, troubleshooting is a different animal. When people make assumptions based on their abstractions, they can waste a lot of time chasing a problem that can't exist, because the abstraction hides several more layers of complexity. Their mental model of the problem relies on a fantasy which they may not even realize is a fantasy.

It's unlikely for someone to accurately diagnose a fault that's several layers below where they're operating. Understanding that their abstractions are abstractions, and knowing when to hand things off to another layer of engineer, is critically important.

I mention it here because people aren't generally tracerouting things unless they suspect breakage somewhere. It's a troubleshooting tool. But people whose mental model of the network _only_ goes as low as the IP layer, are unlikely to do anything useful with traceroute unless the fault also happens to be at the IP layer.

Sounds like fun :)

You should write up a blog post where you had to debug some weird gnarly issues. A lot of us higher up the tech stack are pretty far removed from the signals level issues you're talking about but would love to hear about a low level debug session!

I would love to read such a blog post as well!

Thanks for the clarity in your response. I think we are pretty well aligned.

It is human stacking of the OSI model. The person in the ditch repairing the line probably doesn't care about traceroute, but the person with traceroute probably should care about what happens in the ditch.

The OSI model exists for a reason.

You don't think about the life of the electrons going through your processors when you code.

Traceroute is a view at a certain level of abstraction. It also doesn't tell you if your packet was delivered using ethernet, wifi or a token ring. It just doesn't matter.

It just doesn't matter until it does. It's fine to work at a higher level of abstraction. But people who understand a lower level of abstraction can do things people will call "impossible" with fault injection exploits, rowhammer etc.

The more you know about how something works the better equipped you are to handle things breaking. It's a safe bet that semiconductor physics and the gate-level construction of CPUs isn't necessary to be a good programmer, but not much further up that stack are things like understanding superscalar processor architecture, how caches work, how CPU protection levels work, etc. Knowing about those things, for sufficiently performance or security-intensive applications, can make a ton of difference.

There's an analogy to networking there, too. You don't necessarily need to know how wave-division multiplexing, BGP, or DNS work to communicate over the Internet. For some categories of problems, though, a little bit of knowledge allows you to punch just a bit above your level.

The OSI model hasn't been accurate representation of ip networking since pretty much day 1. It was made specifically for a different protocol, but in the stack we use today some layers are better split up in 2, some protocols exists in multiple layers. It's a nice metaphor but I think it's time to drop it!

https://computer.rip/2021-03-27-the-actual-osi-model.html

Something that I've noticed that somehow ends up lost when people learn "the model" is the encapsulation aspects.

I don't know if it's missing in people's course work or what, but I've had to use http://www.tcpipguide.com/free/diagrams/ipencap.png many a times to explain how stuff like VPNs work, correct statements like "firewalls don't have a routing table, firewalling is layer 4", explain things like MTU and payload size, or why certain traffic doesn't go beyond a broadcast segment normally.

Personally I think this is one of the better visualizations.

To clarify my previous post, asymmetric routing is strictly an L3 behavior, and ECMP routing can also be an L3 behavior where a router chooses one of many equal-cost next hops based purely on data in the IP headers. The exact behavior of course depends on the ECMP load-balancing algorithm in use, whether it's per packet, per destination, or using a hash. And furthermore whether it's strictly IP or if it looks deeper into the packet and uses L3+L4 headers in its decision making.

Both asymmetric routing and ECMP routing are visible from L3. In the latter case, the routing decision can utilize some L4 data, so some L4 frobbing to get useful data points in practice is necessary for useful real-world diagnosis.

I agree with others that the OSI model is a good metaphor and a framework for reasoning about networking, but it is far from perfect, and the reality for those designing and operating network protocols and devices is messy.

MPLS is admittedly invisible and there isn't a thing you can do about it in the same way that you can't expect traceroute to give you a view of the switch ports it went through on a LAN. Of course it is useful to understand and keep in mind the fact that there may be, sometimes huge, gaps in your traceroutes. A sudden huge jump in RTT from one hop to the next can be confusing when trying to understand and troubleshoot a network issue.

CCNA baby

OSI was supposed to be a competitor to IP and Ethernet. That's the reason it exists.

A lot of ISP internal networks is going to be running over MPLS or encapsulation. You're not going to see any of those hops. The packet will just look like it teleported from the CPE to the Internet.

Not sure why this matters for anyone not working for the ISP that those packets go through, and people working there would know about it. And they're mostly completely passive devices like filters.

Traceroute is what we have, and its mostly enough as you can say at which hop things went wrong, then its up the ISP to figure out what devices they need to look at.

It definitely helps to understand it when you are trying to explain to the ISP that it’s their problem. Knowing what you’re talking about will get your issue resolved much more quickly.

That depends who's doing the talking. You're a large business with a dedicated connection and your own network engineering team? Yeah, totally matters.

You're a single consumer, or a small business? Your dialog is limited to "Yes, I switched the router on and off. Yes, my local network works. No, I can reach the website over my phone, this is your problem".

Repeat three times, then get the inevitable reply of "it'll be up again in the next 24 hours".

Unless you're a network engineer, this really doesn't matter.

How does it help the ISP if you're just guessing what they might have between two routers?

as a predominantly wave-purchaser than builder, it matters when it matters and it doesn’t when it doesn’t.

we get by fairly well assuming l1 is working as designed when the light is on and not throwing errors.. at least on day-to-day ops. planning/sourcing is a bit of a different thing.

one note to your point though, back when people were still regularly picking up OC3s or doing frame relay or whatever, it was certainly more day-to-day to understand these things, but cant really fault anyone since most of that junk has gone away and we’re left with the happy ethernet handoff. especially in small scale DC/enterprise. cloud, too, made us care less.

And on the active networking component side of things it doesn't touch on MPLS which also doesn't modify the IP headers. You can enter a network in New York and get MPLS switched across the country via active network devices all the way to California and have it show up as a single hop on traceroute.

The explanation is great for a toy network bu in today's Internet the vast majority of routes are going to be asymmetrical and that requires running traceroutes from both ends and interpreting the results to find the faulty hop.

The author also doesn't cover equal cost multipath (ECMP) which is everywhere. With ECMP you have multiple ports that lead to the same. Next hop and packets are hashed based on some part of the fourtuple, sometimes five tuple including the input Port. In order to track down the faulty link, you need to pro each and every one of the ports which requires that you use a higher level protocol like UDP. Using icmp in this case will not show you an issue some percent of time, providing false negatives which makes it less useful.

Below the IP layer there is always the physical layer somewhere with all the complications.

Traceroute also doesn't show VPNs or other transports.

But it shows the TCP hops and the timing, thus gives me the tools to see who to call or which routing table or wore to analyse or fix. When doing that I got to lok at that connection.

Right, there is also the issue of BGP, Border Gateway Protocol for the core of the Internet?

The smart man knows all of these things exist. The wise man understands when these things matter.

DWDM protected waves, MPLS, etc, are all out of scope when someone does a traceroute, finds their traffic is going to Europe asks me about it, and then I see a local upstream has picked up Cogent as a transit provider, and Cogent is preferring their customer routes over routes from their Tier 1 settlement free peers.

But this article is very much a primer, it's using the unix format of traceroute and talking about ICMP.

You don't know how many times I've seen people claim that they had two redundant connections, when those connections clearly rode the same conduit across a bridge not half a mile down the road. No, I don't care what your traceroute says, the backhoe knows otherwise.

Which command does go beyond the "traceroute" limitations you mentioned ?

This is a significantly better technical presentation on how traceroute works[0]; for example, unlike the illustrations in the linked article, traceroute does not necessarily take a symmetrical return path; the return path is hidden from the client -- the client only sees the forward path.

[0] https://archive.nanog.org/sites/default/files/traceroute-201...

Traceroute doesn't even show you a path. It shows you a bunch of devices that happened to have a packet when its TTL expired. Every item listed in traceroute's output is a different packet and can take a different path towards the destination.

On a different subject, why are people writing blogs about topics that are in the "literature" already?

It’s not guaranteed to be accurate, but tracing using the UDP/dublin strategy with a fixed dest port and varying src port per round can help to identify and visualize valid ECMP flows. I recently wrote some guidance [1] on using Trippy in this way.

[1] https://github.com/fujiapple852/trippy?tab=readme-ov-file#ud...

Once I used iperf3 with 100 different udp streams/srcports to troubleshoot an issue, a small % of connections had >90% packet loss and this caused the connection pool of this service to fill up until it had only failed connections waiting to time out or going extremely slowly. ISP told me it was a broken linecard in a router, so packets were being dropped/corrupted on the backplane between the linecards.

Traceroute and mtr didn't use enough ports to show the issue clearly.

I used this to identify some intermittent loss in one direction from I think Kyrgyzstan to the UK, There was a claim that sometime flows were fine and sometimes they weren't. Problem was an ECMP in Pakistan or Iran which was using udp port numbers to balance the flow. Could consistently get 25% loss with certain numbers and not with others.

In English, route is synonymous with 'path'.

Feel free to submit a PR if you disagree:

https://man.openbsd.org/traceroute

This isn't an English language question. The technology in question doesn't guarantee a singular route or a path to a destination. Subsequent packets from an originating host to a destination host can follow different paths. Even during an activity such as a traceroute. A packet that reached its TTL at what is presented as hop 12 didn't necessarily traverse previous hops displayed by traceroute.

You'll learn pedantry gets tiring when you get older. How traceroute is understood and displayed to the end user constitutes a route. If you have an issue with how the developers of traceroute framed it, that's on you.

OP was the pedant in this case, unfortunately (for you).

Hey! What literature would you recommend for learning more about this stuff (routing, firewalls, etc)?

I'm learning as I'm going and my goal is to come up with a way to build secure/beyondcorp/zerotrust-style networks using only existing tools and established tech, unless absolutely necessary.

Tools I'm using include wireguard, iptables, nftables, nginx, ping, traceroute, and nmap.

Which articles, books, authors, or repos must I read and which concepts must I understand?

Trying to find real knowledge about this stuff online is a nightmare!

Related threads with recs:

Ask HN: Good book to learn modern networking? - https://news.ycombinator.com/item?id=38918418 - Jan 2024

Ask HN: Best Books for Modern Computer Networks - https://news.ycombinator.com/item?id=39501933 - Feb 2024

Ask HN: What is a good book to learn about the network stack? - https://news.ycombinator.com/item?id=18506651 - Nov 2018

Nothing can be assumed to be take a symmetrical return path. Even if I send a normal packet to some server on the Internet, the server's reply may or may not come back using the same path. As a former Google employee, I remember vaguely that this is something Google occasionally does on purpose.

It always amazes me in-depth HN gets on pretty much any subject, yet the most basic cursory network page gets massive response

Is there simply a total lack of understanding of how networks work in the tech community?

I'm often tasked with explaining technical concepts to people with entry-level knowledge so simple, clear documents like this are very helpful resources.

Would a “a for loop in node js” or “serving an webpage in Python” get so much attention?

Make one, and lets find out

People who want to learn how ISP networks work can go and join DN42, a LARPing copy of the Internet.

Networking is a black box for many developers.

Why would it be any different if you're developing a CRUD application with a bunch of SREs / datacenter engineers on the line? They are very good at abstracting networking away from software engineers.

Yes. I've worked across a range of companies for small startups to Fortune 500s for nearly 30 years, interviewing for positions from system administrators to dev ops to programmers, beginners to architects. There's just not a lot of generalists out there. It always surprises me how far folks can get in their careers with a very narrow knowledge base.

A couple years ago I was bringing a Java programmer up to speed on some C code just to learn they had no idea what a call stack was or how it worked. They were familiar with the stack as a data structure, but had no idea how a CPU worked or that it has a stack pointer register.

As long as the abstractions don't leak, I guess everything is fine. I mean, I can't really tell you at a physics level exactly how semiconductors work. At best I can hand wave an explanation.

That said, the lack of even hand waving knowledge about how the internet works among professionals who use it every day continues to surprise me.

Yes. Learning to "program" has never been easier, but nitty gritty stuff like networks and routing is still much more technical than the avg SWE

Wait until DNS comes up again. I don't know how many DevOps, SRE, Senior Architects, etc. that I've worked with that just don't understand how names are resolved. There's a wide array of specialists that don't understand general fundamentals.

I also see this. Getting an MX record or TLS cert deployed is often a source of confusion, even with "seniors.'

Personally, I understand the fundamentals until they all change and DNS has had a lot of outgrowths relating to encryption, Email spam prevention, IPv6, etc that have made things a lot muddier for me over time.

As someone with a good deal of networkimg knowledge, yeah, most people don't have very much.

You can do a lot without knowing anything about routing, and very little about protocols. And all those network people always seem grumpy, why learn about it if you don't want to be grumpy. ;p

Edit to add: how networking works in reality and how networking works in books and coursework tends to be quite different as well. GeoDNS and Anycast IP are a big conceptual jump for people who only understood networks from coursework. There's also a common assumption that routing protocols favor least distance routing which is certainly not the case. I may be obsessed with path MTU problems, but that's a real hard concept to explain to experienced tech people with slim networking knowledge. Also, a lot of people seem to believe the OSI model is fact and not a model of the world; it's not unuseful, but there's lots of hardware and software that operates at more than one OSI layer simultaneously and people stuck on a rigid model can have an incorrect view of the world which impedes their problem solving.

Yes, I started my career in big tech and everything was so abstracted away I never really knew how it all worked as a junior. When I went to a startup and it was not longer all abstracted away I learned a ton about how it works (we were working on something cybersecurity related).

When I went back to big tech it really helped me out just knowing basic things about networking. For example, my manager joked that people should have to take an intro to DNS as part of their on-boarding to big tech.

I don't fault people for not knowing either, it was just never really a priority and the point was you shouldn't have to care. You should become an expert in your business domain and spend time solving business problems. Whether or not that was the right approach is up for debate though.

Absolutely. I'm a software dev and have been doing this for many years, but never had to think about the network much. I recently went down a BGP/AS rabbit hole and found it _fascinating_, and that's just the tip of the iceberg - go up or down the OSI stack and there's all kinds of stuff I didn't know about!

If the tools to set up a network actually did what I asked them to, this page would be simple. In another comment you compared it to a node.js script.

The difference between networking and programming is when you call a function it will actually do what you say, every time. In a network you can waste hours and get absolutely nowhere.

I've been in IT and software for 25 years at various levels both as a job and a hobby, and I can confidently say that networking is the biggest waste of time, most frustrating, and least rewarding aspect of working with computers. No matter what I try to set up, it never, I mean never works the way I want it to. And I've done plenty of projects to get a healthy sampling, some of them include NFS/CIFS, RDP, DHCPD+PXE, VPN, IPFW, BIND, iptables, SMTP/IMAP. Working with anything networking is absolute hell. Every single time. Guaranteed.

> NFS/CIFS, RDP, DHCPD+PXE, VPN, IPFW, BIND, iptables, SMTP/IMAP

The hardest part of configuring the services you list is not the networking, but all the non-networking topics you need to be familiar with: zone file syntax; cipher suite selection; OS-specific security policy and permissions models; PKI administration; PXE ROM peculiarities, etc.

Once you're confident with arp, addressing/subnetting, routing tables, and a few other tools like netstat and `dig` you can quickly eliminate the network as the cause of the issue and focus on grokking the application itself.

Then we have different definitions of the word. That's fine, the top comment at the time of my posting made it sound like "networking" was understanding the interactions of optical repeaters on the glass cable. To me "networking" means actually using network applications for my benefit.

Yes and that's why hiring an experienced network engineer is incredibly goated

These days, us network engineers are more often using mtr to explore networks.

https://www.cloudflare.com/learning/network-layer/what-is-mt...

Somewhat related is this cool project by hack club [0] (not affiliated just a big fan!):

how-did-i-get-here: "A tool/website/article by @kognise about how routing on the Internet works."

Site: https://how-did-i-get-here.net/

Github: https://github.com/hackclub/how-did-i-get-here

[0] https://github.com/hackclub

Transport networks are often mpls, SR, whatever, and this approach reveals nothing inside the SP ASes for the most part.

I miss traceroute. It seems like more and more, the linked page is hypothetical, and what I get in practice is a couple of hops, an arbitrary number of "* * *" lines, and maybe the last host or two. It's so nice when it works.

    $ traceroute youtube.com
    traceroute to youtube.com (142.250.114.91), 30 hops max, 60 byte packets
     1  10.202.10.88 (10.202.10.88)  0.384 ms  0.356 ms  0.342 ms
     2  10.202.35.103 (10.202.35.103)  36.386 ms  36.370 ms  36.325 ms
     3  10.202.32.4 (10.202.32.4)  36.301 ms 10.202.32.3 (10.202.32.3)  36.301 ms 10.202.32.4 (10.202.32.4)  36.287 ms
     4  10.202.32.2 (10.202.32.2)  0.764 ms  1.279 ms  1.250 ms
     5  lo0-0.gw2.rin1.us.linode.com (45.79.12.102)  0.610 ms lo0-0.gw1.rin1.us.linode.com (45.79.12.101)  0.663 ms  0.650 ms
     6  ae62.r22.dfw01.ien.netarch.akamai.com (23.203.147.40)  0.986 ms  0.881 ms  0.837 ms
     7  72.14.204.254 (72.14.204.254)  1.164 ms 72.14.198.98 (72.14.198.98)  1.153 ms 142.250.47.248 (142.250.47.248)  2.926 ms
     8  * * *
     9  209.85.251.24 (209.85.251.24)  1.082 ms 142.251.71.114 (142.251.71.114)  1.302 ms  0.950 ms
    10  142.251.234.214 (142.251.234.214)  1.267 ms 216.239.58.16 (216.239.58.16)  3.296 ms 142.251.66.192 (142.251.66.192)  20.440 ms
    11  108.170.228.86 (108.170.228.86)  1.161 ms 108.170.228.82 (108.170.228.82)  4.548 ms 108.170.228.81 (108.170.228.81)  1.746 ms
    12  108.170.231.7 (108.170.231.7)  3.484 ms 108.170.229.87 (108.170.229.87)  3.071 ms 142.251.70.211 (142.251.70.211)  2.938 ms
    13  142.250.236.158 (142.250.236.158)  2.298 ms 216.239.51.220 (216.239.51.220)  27.388 ms 108.170.233.60 (108.170.233.60)  2.001 ms
    14  142.250.224.27 (142.250.224.27)  2.085 ms 142.250.224.25 (142.250.224.25)  1.994 ms 142.250.224.23 (142.250.224.23)  2.558 ms
    15  * * *
    16  * * *
    17  * * *
    18  * * *
    19  * * *
    20  * * *
    21  * * *
    22  * * *
    23  * * *
    24  rr-in-f91.1e100.net (142.250.114.91)  1.997 ms  1.981 ms 1.967 ms

Just an example. There's still a lot of stuff that works, but it just seems to me more and more often that when I have an actual problem I'm going to see a lot of * * *. So many things turning the requisite packets off.

(To save people excited to see if I've leaked something important some time, this comes from my Linode-hosted website's box, not my home connection.)

traceroute noob - what do the * * * mean? A host is choosing not to participate and dropping data or something?

Traceroute works by observing Internet Control Message Protocol (ICMP) time to live/hop count exceeded messages and noting the source IP of those messages. There's plenty of resources you can search on to learn more about this. When you see * * *, it means that for that given TTL value, no corresponding ICMP messages were returned to the source.

It could be because they were filtered at the boundary of the network. If source IPs are private (RFC-1918 or carrier grade NAT space), those ICMP messages should get dropped. This is Best Current Practice (BCP) 38, if you wanted to read more about it.

Ever want to know why IPv6 is so important? This is another reason why. Troubleshooting networks without globally significant IP addresses on the intermediate hops is a real pain.

Great info, ty!

To be fair, that entire traceroute never touches "the Internet". Everything after hop 6 is inside Google's backbone network, and everything before hop 7 is inside Linode/Akamai.

So it hits the Internet once between Akamai and Google I guess?

I mean, KIND OF, if you squint. From the trace, Akamai and Google have a direct peering link (hop 6 looks like an Akamai edge/peering router -- Akamai owns Linode so that makes sense -- and hop 7 is a Google peering router).

For it to be hitting "the Internet" you'd expect to see hops on a Tier 1 carrier like AT&T, Lumen, etc. and/or an ISP like Comcast, Spectrum, etc.

It's still the internet, tier 1 carriers aren't special, they're just big.

Back in the day, all I wanted is to be able to do like Natalya Simyonova in Goldeneye, in the train. I recall during the late 90s there was a (Windows?) programme I knew that did this, based on traceroute/tracert I would suppose. Just a small memory to share here, maybe others had it too :)

On the other hand traceroute isn’t real[0]

0: https://gekk.info/articles/traceroute.htm

> It is, generally speaking, not possible to call AT&T and say "Hey, when I try to ping one of your subscribers in California from a Level3 circuit in New York, I'm hitting a routing loop."

I’ve reported similar things to my isp and they’ve changed their lock prefs to send the traffic via a different peer and bypass the problem.

There are more private subscribers that pretend they know what they’re talking about than actually do. Probably makes sense to ignore a good proportion of the nonsense people throw at ISP L1 teams.

this article is informative but misleading. i do the testing at a big ISP (your packets go over us for at least something). we test traceroute. we complain to vendors when traceroute doesn't work right. we do investigate weird traceroutes.

> your packets go over us for at least something

I'm in South East Asia -- are you sure?

I think he meant “us” as ISPs in general.

No I think he/she was referring to his/her "big ISP".

It's very likely my packets don't go through it.

its not just about what you're connected to, but what the other side is and how the things work out to get it in between. SE asia is an important area for some large ISPs.

> Traceroute is a filthy hack

This is said like it is a bad thing…

Trace route has such a nice promise of untangling the internet into its constituent parts. We used it last year in an installation where we physicalised the internet as marble run. You would create a packet for any website, and the visit all the hops one-by-one across their various locations[1].

[1]: https://youtu.be/9uIs0sh4iYU

building a visualization system for traceroute in the mid 90s was how i taught myself java.

traceroute itself was wrapped by a perl rpc wrapper and then a perl www cgi script would take a list of hosts and then reach out to all of them to ask them to traceroute each other, then a java applet would render an interactive graph that you could rearrange as you saw fit.

interesting learning: internet routing can be asymmetric!

> But then, the path is broken, and we can’t reach the destination computer.

No. This is not what this output means.

People should not read more into what traceroute says than what it actually does.

* * * means that traceroute did not get an ICMP time exceeded for that probe. that's it. It says nothing about if you can actually reach the destination.

Seeing the title reminded me of "Warriors of the Net".

We were genuinely told, as a class, to watch this video to learn how the internet worked.

https://www.youtube.com/watch?v=RhvKm0RdUY0&themeRefresh=1

Good times!

Once you know how routing is supposed to work a nice homelab challenge is to set up a redundant route between your PC and the internet, like going through your NAS or something, and watch it failover when you unplug your network cable.

Better to use IPv6 in all examples. Time to get used to it instead of IPv4.

If the destination machine has disabled ping, what response is recieved?

nothing from that machine, but the way the ttls work, it doesn’t affect the responses from routers along the way. Same if the destination doesn’t exist at all.

Totally untrue. Network admins will often disable traceroute responses because security.

Edit: the less someone knows about your internal topology the better. Security through obscurity does work.

~~ Security through obscurity does work.

As a layer.

Windows disables ping by default for what I'm guessing is security.

I was answering a question about what happens if it is disabled on the destination machine. The destination machine has no say over any other device along the route.

“ping” is ICMP ECHO_REQUEST and ICMP ECHO_RESPONSE. Traceroute uses ICMP TIME_EXCEEDED. So blocking only “ping” will not affect traceroute. And if you block all ICMP, you break your own internet: <http://shouldiblockicmp.com/>

The number of companies that I've encountered that wholesale block ICMP because "security" is painfully high.

Tracert is more powerful than you think:

There are a lot of good talks on Tracert.

This one is pretty good:

https://www.youtube.com/watch?v=jGYAW5z6BJc

The article OP links to is only talking from the perspective of an internal network. Tracrt in a 10. network is boring info.

In the vid I posted, he gives you really good common advice about using tracert to show you actually the physical layout of the path:

https://i.imgur.com/LxN9Mr4.png <-- Using the DNS name of the router is great, because us network nerds like to use naming conventions in a graph format: so you can tell that its edge router number N at location B in City X and using tracert - you get to see the national networks the packet hits.

THen by seeing the carrier, you can also see where there is not just a change in carrier, but also that indicates that at that location is a datacenter....

You can go onto DatacenterMaps and find out who/what/where a DC is....

(There is a really exceptional tech talk on tracert thats quite long that goes into bitlevel detail of weaponized tracrt - but I cant find it)....

---

WRT DataCenterMaps -- There was an HNer that posted about mapping nuclear facilities (active and decommissioned) - and by using his map, the UnderSeaCableMap and the DataCenterMap - then by looking at shipping supply-chains for components used_by/made_by/received_at companies that are either Data Centers, or NVIDIA - we could track where large scale AI componentry is being installed into what data centers, which are fed by which Nuclear Power Plants, who have to report on their Consumption Graph - and which Cable Infra is likely feeding each DC.

We can see where AI traffic flows - and by using tracert at a deeper level - we can see exactly the AI's Physical NeuroNet' - and find a way to measure its power consumption and physical footprint.

---

HNer @externedguy "..built interactive map of active & decommissioned nuclear stations/reactors"

https://news.ycombinator.com/item?id=41189056

(I correlated the Nuclear reactor locations with DataCenters, undersea cable endpoints (which will be near both nukes and datacenters)

As they could be layers - and we track shipments and we can see where AI consumes:

---

...if we add the layers of the SubmarinCableMap [0] DataCenterMap [1] - and we begin to track shipments

And

https://i.imgur.com/zO0yz6J.png -- Left is nuke, top = cables, bottom = datacenters. I went to ImportYeti to look into the NVIDIA shipments: https://i.imgur.com/k9018EC.png

And you look at the suppliers that are coming from Taiwan, such as the water-coolers and power cables to sus out where they may be shipping to, https://i.imgur.com/B5iWFQ1.png -- but instead, it would be better to find shipping lables for datacenters that are receiving containers from Taiwan, and the same suppliers as NVIDIA for things such as power cables. While the free data is out of date on ImportYeti - it gives a good supply line idea for NVIDIA... with the goal to find out which datacenters that are getting such shipments, you can begin to measure the footprint of AI as it grows, and which nuke plants they are likely powered from.

Then, looking into whatever reporting one may access for the consumption/util of the nuke's capacity in various regions, we can estimate the power footprint of growing Global Compute.

DataCenterNews and all sorts of datasets are available - and now the ability to create this crawler/tracker is likely full implementable

https://i.imgur.com/gsM75dz.png https://i.imgur.com/a7nGGKh.png

[0] https://www.submarinecablemap.com/

[1] https://www.datacentermap.com/

----

And 8 months back I posted:

In the increasingly interconnected global economy, the reliance on Cloud Services raises questions about the national security implications of data centers. As these critical economic infrastructure sites, often strategically located underground, underwater, or in remote-cold locales, play a pivotal role, considerations arise regarding the role of military forces in safeguarding their security. While physical security measures and location obscurity provide some protection, the integration of AI into various aspects of daily life and the pervasive influence of cloud-based technologies on devices, as evident in CES GPT-enabled products, further accentuates the importance of these infrastructure sites. Notably, instances such as the seizure of a college thesis mapping communication lines in the U.S. underscore the sensitivity of disclosing key communications infrastructure.

Companies like AWS, running data centers for the Department of Defense (DoD) and Intelligence Community (IC), demonstrate close collaboration between private entities and defense agencies. The question remains: are major cloud service providers actively involved in a national security strategy to protect the private internet infrastructure that underpins the global economy, or does the responsibility solely rest with individual companies?

(There was talk on this topic recently in the news)

EDIT:

If you like this sort of networking - then courses/material like this are great:

https://www.youtube.com/watch?v=Ih3KgQnT6T0 <-- network recon, scanning, countermeasures - failryl vanilla, but concise.

---

I still cant find the defcon-style talk that really dives into tracert sorcery....

seems like clickbait

traceroute uses udp to a high-port for discovery, not icmp-echo.

traceroute supports multiple protocols, not just UDP and ICMP.

true, the type of packet does not really matter.

my apologies if this misconstrued my point about the lacking quality of the article.

Windows tracert uses ICMP

Source: https://support.microsoft.com/en-gb/topic/how-to-use-tracert...

> The TRACERT diagnostic utility determines the route to a destination by sending Internet Control Message Protocol (ICMP) echo packets to the destination

true, most computers don't run windows though.

anyways, the type of packet does not really matter.

I agree that this doesn’t seem to have been written by someone that actually understands the topic.

Even the network diagram at the beginning is not very good. Can you create network architectures where 10.0.0.1 and 10.0.0.2 are not layer 2 adjacent? Yes, but they’re fairly complex and would imply that a lot of other necessary information is missing from the diagram. And should you use such an architecture as an example to explain traceroute? Absolutely not. It’s hard to imagine someone with even a CCNA level understanding of networking coming up with this.

/31 linknets are common in the ISP world, it might even be /32 loopback of those devices, who knows.

I do agree that its bad but not for the same reason. On the diagram it looks like router IPs are their loopback IPs rather than the link IP, as we know a traceroute response comes from the interface the packet came in and that IP is used in the response. Seems like the creator tried to simplify the diagram and excluded linknets but made it more confusing instead.

Glad I'm not the only one who was bothered by this. No subnet masks in the diagram, and I'm supposed to assume that 10.0.0.1 and 10.0.0.2 are not in the same broadcast domain. So my default gateway is what, 10.0.0.0, and that router has a route to 10.0.0.2/? with a next-hop of 10.1.0.5?

Wackiest network I've seen in a while, and I've seen some real winners.

As a firewall-admin this comment really annoys me.

Heres why: Lots of calls from people complaining their application is not working because the network is broken. Look! at! this! traceroute! supporting! my! complaint!!!

Yeah, it is because udp from any to any is blocked on the firewall for good reasons while icmp traceroute is open. And be carefull because -I specifies the outgoint interface on some os, not the use of ICMP.

most applications don't run over icmp so it is indeed sensible to not use it as a default.

Reply on news.ycombinator.com