Unit testing a TCP stack | Hacker News Reader

Unit testing a TCP stack

snellman.net

89 pointsdjoldman7 comments

Google has a pretty mature TCP testing framework, called packetdrill: http://static.googleusercontent.com/media/research.google.co...

It supports all the features you need, from packet structure to timing: https://code.google.com/p/packetdrill/

It still treats the whole stack as a blackbox. Necessarily the effort to cover important boundary cases throughout the TCP stack can get overwhelming.

Honestly, I've found black box testing at public boundaries of a system gives the best payoff vs development time, especially if you can make them run quickly.

Most issues seem to be catchable with this, and way too many unit tests end up becoming 'change detector' tests, where you test tautologies about implementation details.

Black-box testing as a first move is often a good move, IMO — but it's important to accept that at some point you're going to stop hitting so many edge-cases in the implementation that ought be tested. Some of those edge-cases should probably be tested with black-box tests, and some should probably be tested with unit tests.

One has to consider the performance cost of black-box tests, though. It will almost always be higher than that of unit tests, simply because more code will get run per test (this is especially true of network stuff, as even going through the loopback device is comparatively insanely expensive). That said, in some cases, it may well still be the right choice.

> black box testing at public boundaries of a system

But that sounds more like integration testing, not unit testing, doesn't it?

I watched the Usenix talk on packetdrill, and while it's a really neat tool it seems to me that it doesn't really support timing very well.

The tests are run in real time - running 650 tests take almost half hour, which would normally be considered completely unacceptable for any kind of TDD. The timing can also vary from one run to another, so all timestamps have a few milliseconds of wiggle room automatically, and a small proportion of runs (1/2500) still fail randomly due to timer inaccuracies. And unit tests that fail non-deterministically are really unpalatable. Is the talk outdated re: these issues?

Great hack despite that, and I would certainly have loved to write a scripting language for expressing tests rather than use C++. And the packet syntax they've chosen is probably superior to what I ended up with. But what I describe in the post dates back to 2011, so packetdrill wasn't yet around for us to take inspiration from :-/

Very nice. The fact that the TCP stack isn't in-kernel isn't all that necessary either. You can perform the same abstraction for an in-kernel stack as for an in-application stack, though it can often be harder to disentangle a subsystem that wasn't built from the start with that kind of separation in mind.

Unfortunately I can't share them, but I have done exactly this sort of thing in embedded projects at work - compile the same C code (which is coded in a style that tries to minimize if not eliminate undefined and implementation-defined behavior) to run on the host that builds it, under a Haskell test harness that tests behavioral aspects using both unit testing and Quickcheck property testing, with Haskell code simulating external stimuli such as interrupts, low-level behavior of storage subsystems, mcu resets, etc.

Testing low-level C code, even code that interacts directly with hardware in unprotected domains, is absolutely possible. And also extremely valuable. The usual caveats about test correctness are even more important, though - there's a much higher burden to ensure that your simulated hardware is both accurate and comprehensive in the behaviors it can simulate.

Reply on news.ycombinator.com