Years ago I worked for a company who was building a piece of networking hardware, and we had bought some code that implemented various well-known routing protocols. This included the TCP/IP layer, but it was just the BSD stack.

Our dev hardware had a boot loader that would use FTP to download the most recent images to finish the boot. After we cut over to this code, we started having an occasional problem where this download would fail -- the box would send a spurious RST and kill the connection. I was one of the protocols people, and it ended up on my plate.

I spent 3 days staring at tcpdump, adding enormous piles of debug traces, and finally going through the code line-by-line with a copy of the Stevens book open on my lap. Eventually I found the problem - they had made one single change to the BSD code, and in so doing introduced the bug. It was compiler-dependent.

Writing this code correctly is hard.