A Practical Guide to Correctly Troubleshooting with Traceroute (2009) [pdf] | Hacker News Reader

A Practical Guide to Correctly Troubleshooting with Traceroute (2009) [pdf]

nanog.org

150 pointskercker10 comments

Reminds me of back when I was a kid in NYC around 2001, and first learned about traceroute and routing playing around with Linux. Late one night my Comcast Internet went down. So I called support, only to be given some message that it consumer support was closed. Did some basic ping and traceroute troubleshooting and discovered that the first hop out of Comcast to Cogent was down.

So being a foolish and persistent 18yo, I called directory assistance and sure enough Cogent NOC was listed. So I called them pretending to be a Comcast support employee. No questions asked. I told them the router IP that was down and they said they'd look into it. 20 min later, my connection to the outside was up. Probably fixed the Internet for several million people in NYC. The foolishness of youth has its perks.

> So being a foolish 18yo I called directly assistance and sure enough Cogent NOC was listed. So I called them pretending to be a Comcast support employee. No questions asked.

How in the world did you find the correct phone number? And how did you pretend to be an employee? (Did you say "this is [random name, or maybe your own name] calling from Comcast?" or something?)

The world was different back then. I remember doing a Whois lookup and calling the technical contact of a major national isp. He answered the phone personally. I told him one of his DNS servers wasn’t responding to queries. He thanked me and fixed it.

It was easier in those days to find NOC phone numbers and network engineers generally like to fix problems.

I did network event correlation in those days (using prolog and Perl to convert hundreds of thousands of log entries and events to actionable alerts), and would call various NOCs all of the time, usually to let them know that their device was spewing alerts or doing something dumb. There was very little in the way of authentication. Once you crossed the line to ask for things vs fix things you’d get more questions.

> It was easier in those days to find NOC phone numbers and network engineers generally like to fix problems.

Yes, it is super hard now. It took over 5 seconds to find the Cogent support numbers:

http://www.cogentco.com/en/customer-service/support-desk

It if hadn't found anything after 10 seconds, I would have given up and gone back to watching Youtube.

And if I call those numbers I get the people in charge of running the network (which the conversation was about), or a helpdesk that can't do anything directly?

Honestly memory is fuzzy now. But I think I called the 800 directory assistance. I don't even know if it works anymore, but it was big back then.

Not sure when the Puck list started, but when I was in the ISP space, I used to it contact other NOCs frequently.

https://puck.nether.net/netops/nocs.cgi

Monkeying around with traceroute can be all kinds of fun.

Some years back i was having some issues reaching anything beyond the neighboring nations, so i ran a traceroute on one of the problem sites and noticed the routers were labeled by city.

Sure enough, fire up the local news and i read that there were some severe weather going on in the area of the outage...

Presentation video: https://www.youtube.com/watch?v=4dUqVlZ6trU

Reply on news.ycombinator.com