Tracking NFS problems down to the SFP level

I no longer have access to it, but I wrote a tool to find these types of problems at my last job. It didn't seem generally applicable enough to try to get it open sourced (and I didn't want to polish it enough for that either).

The key insight is that LACP is almost always configured to use do a hash of { Source IP, Dest IP, Protocol, Source Port, Dest Port } so that packets from each TCP and UDP flow will always be sent on the same individual link. (this is directional though, so it may go from peer A to peer B on cable X and from B to A on cable Y).

So the way to confirm a broken link is to connect a bunch of UDP flows (on different ports) between peer A and peer B, send data, and measure loss (and/or delay!). If you see zero loss / uniform delay, either none of your flows cross the broken link or the problem isn't a broken link or the rate of issues is too low to detect. Once you've found a broken flow, you can use a 'paris traceroute' tool to confirm the IP routers it's between. Paris traceroute holds the UDP source and destination ports fixed, so the route on LACP should say the same. I contributed support for this in mtr, but I'm not sure if it still works; if you see 100% packet loss with mtr in fixed udp port mode, send me an email (address in profile) and I'll ask you for data and try to debug.

Once you narrow down to the two routers the link is between, it should be easy enough to confirm. Usually through link quality counters, if not, through just pulling links and seeing if things get better or worse.

If you have a long network path, and most links are LACP, the total number of possible paths between two peers gets large, and there's a chance that you might not be able to survey them all from a pair of servers; so you may have to try a few different hosts.

You can find packet loss, but also congestion/buffering this way. In an ideal world, all the link error counters would be collected and anomalies would be addressed, but it seems that doesn't always happen.

Given enough of these fault analysis articles, I wonder if it’s possible to compile them into some kind of decision tree-like interface where you can describe your problems and have it guide possible failure scenarios (plus diagnose steps). Would be cool to collect all of this knowledge beyond google, as this is the type of stuff where Google often breaks down.

So one comment here on ingress vs egress. The type of forwarding a switch does, store & forward vs cut through can effect where you see the errors. In S&F the switch buffers the full packet before it switches it out. If the FCS is off at that point it drops the packet. In CT the switch will start forwarding a packet before it has received the full packet, so a corrupt packet will be seen on the other side even several hops away if the full chain is CT. Some switches do some magic on the fly and if they ID the packet as bad sometime before the final data they will purposely corrupt the packet so the other side see it. To be clear I am speaking in general terms and there are more details here. Kalpana did the first CT switch back in 90s. Cisco picked them up. CT sort of died there until Fulcrum ASICS in the early 2000s. Arista’s 71xx line used their Bali chips. CT hit mainstream again with the release of the Broadcom Trident which was used by every major player (Arista 7050, Cisco Nexus 3K, Juniper QFX, etc.)

This seems like an issue that could have been resolved a lot quicker if they were doing network monitoring on the host size and the switch side.

Ideally you would be able to spot a large amount of link errors on a port/switch/host, and fix it before it becomes a problem.

Ha, funny to see the resolution to my bug report (among others) on front page of hacker news, very grateful of SRCC’s work to resolve!

My meta-fix for such a thing would be to hack up a script to start putting these interface error counters into prometheus, and then alerting on a spike above some threshold.