Comment by js2 | Hacker News Reader

Comment by js2 | Hacker News Reader

I probably would've started at the TCP layer only because I've been bitten at that layer many times and it always has these sorts of strange symptoms. Some examples:

1) Connections hanging over a frame relay network that one day started dropping packets over a certain size. Work-around was adjusting the MTU until I was able to convince the frame relay network operator that something was broken in their network. Initially it was confusing because an interactive telnet session over the network would work fine till you did something like "ls -l" or tried to read a man page which generated enough text to send a full size packet, then the connection would hang.

2) Unable to reach a Verizon e-mail paging gateway but only when connecting from a Linux box. An OS X box on the same network as the Linux box could reach the gateway fine. Turned out Verizon had a firewall rejecting connections where the ECN bit was set. Linux was setting ECN, OS X was not.

3) Solaris box A could initiate a connection to box B, but not the other way. After A talked to B, B could then talk to A, but only for a short period. Someone had deleted A's own MAC from A's ARP table, so A wasn't replying to ARP requests for itself. But if A connected to B, B would keep A's MAC in its own table till it timed out after which B couldn't initiate connection to A any more.

4) All manner of misconfigurations over the years where you learn to recognize the symptoms: misconfigured netmask size; misconfigured duplex; duplicate IP address on same network. You rarely see these any more.

5) The infamous 500-mile e-mail. :-)

6) And my favorite - https://www.pagerduty.com/blog/the-discovery-of-apache-zooke...

I agree. I'd probably look at it from TCP layer shortly after initial failures to diagnose if not from he start. Especially when dealing with communication between a cloud provider and on-prem gear and infrastructure. However, it's tempting to exhaust all other avenues depending on how likely the on-prem ops folks are to punt the issue.

I actually did look at the TCP layer early on. However, I didn't pay close attention to the TS Val. From the packet dumps, it just appeared that the TCP window had stopped sliding. I couldn't conclude that NSOC's router was at fault.

Getting NSOC on-board is a big deal. After all, they deal with the entire VA network with 100,000+ employees. If you think about it from their perspective, why is USDS' TCP connections so special?

Network level troubleshooting is incredibly difficult, especially for individuals who don't have a networking background. Even showing someone how to read wireshark often isn't enough.

I just wanted to politely point out though, in this case, I think there should have been an indications of a network failure in this analysis early on, from the standpoint that TCP frames were sent to the server which were not acknowledged. This would depend on the point where you capture the traffic naturally, but the lack of acknowledgement would be a strong indicator that traffic is not reaching the server, or that replies are not reaching your capture point.

So while the TS Val may be the cause of the drops, I think the packet drops should have stood out when seeing the traffic being black holed, and likely the same segments getting re-transmitted continuously.

And for anyone out their who thinks this is easy to catch, I'd say this is very easy to miss, because you need to have a good understanding of how TCP works in the first place, to know what not working looks like.

True, but Wireshark will highlight dodgy TCP frames (retransmits, dups, etc) which should give a small clue to look further. I agree that it is necessary to understand how TCP works (or have access to someone who does) in order to run Internet services.

I applaud you for not putting the obvious work-around in place:

- Inserting "sleep 300" into the startup process.

- Adding a cronjob to reboot the servers once a week.

I kid, but I'm sure we've all seen hacks like this.

Reply on news.ycombinator.com