Hacker News Reader

Comment by anaisbetts | Hacker News Reader

Unit tests are poorly suited for kernel code. The *vast* majority of production failures in kernel code are due to race conditions and other timing-specific issues, which unit tests are extremely poor at reproducing in a way that mimics how the code is used in real life

Another huge source of issues is due to hardware doing something unexpected/not-to-spec - another thing that unit tests would very poorly verify given that, any unit test will simply reproduce how the developer thinks some piece of hardware works, rather than what it does in real life

Production kernels are usually better served by long-running stress tests, that try to reproduce real-world use-cases but inject randomness into what they are doing. And indeed, both NT and Linux kernels are extensively tested in this fashion

View on HN

As a HW guy this seems insane to me. Just because it's hard to do doesnt mean you can't randomise and try to create and catch failures. Might not catch 100%, but better than not trying. HW testing pre silicon is impressively comprehensive as the cost of respinning a chip is in the order of 10s of millions, so the industry found a way. While software is less costly to edit, the cost of updating and rolling out is essentially huge, but uncounted. Finding bugs by extensive testing, even for difficult to catch items, is inherently the right thing to do

As HW guy you are making the hardware so you can make a mock of it relatively easily, that is not the case if driver is essentially reverse-engineered out of how windows driver worked.

And many other drivers are "some vendor made it, and they won't tell us how the chip actually work, just give us driver code".

Then as other people mentioned hardware isn't exactly 100% predictable, HW bugs happen, firmware upgrades change how stuff works etc. so your test suite might be entirely unable to catch that.

I know hardware isn't 100% precitable, which is why we expliclty try to test that behaviour through randomized testing, formal proofs, etc. I'm not mocking anything, I'm just saying it's possible to attempt to cover difficult to create scenarios.

I believe the parent was referring to Test mocks[0], not implying you mocking anything.

0. https://en.wikipedia.org/wiki/Mock_object#Use_in_test-driven...

I see, thanks. Well we do have events in HW that are non determistic, i.e. metastability due to async signal synchrinzation, but that is something we can and do model as best we can

As a combined hardware/software person, the testing for both types of systems is very different.

In hardware, you often have rigid boundaries with well-defined behavior, and almost no state in each unit. A big part of hardware verification is just testing that you did the basics of the interface right and that you won't deadlock anything. The rest of it is testing the behavior of your nearly-stateless thing.

Software testing tends to involve exponentially larger state machines, and fuzzier abstraction boundaries. The basics are mostly done for you, and you tend to do complicated logic that holds and manipulates a lot of state.

"Fuzzing" in software is very similar to constrained random testing in the hardware world, but it still doesn't catch a lot of basic issues due to the size of the state machines involved.

The testing philosophy you need to apply is not the same at all.

It's interesting you have the perspective that hardware has realtiveley few states. It can be the case, but there are very complex interactions between functions also. A million gate design is realatively modest by modern standards, and there can but tens to hundreds of interacting state machines in that. Of course to mange this we do try define clean boundaries and interfaces between blocks, but this clean approach can be taken in SW also. Fuzzing is a bit different in that actually tries to execute different code paths by changing the input, so it's more of an exploration of the code state space. Constrained random is more trying to create inputs that randomly cover the valid input space. It does not introspect the internal state to determine the next input. Perhaps thats an enhancement hardware testing could take from the SW world.

I have seen functional-coverage-driven constrained random verification of hardware, which does what you describe. Someone beat you to it!

They do apply a lot of fuzzing tests actually and the kernel maintainers also contribute state-of-the-art dynamic analysis tools like UBSAN to check for undefined behaviors or leaks.

I think you have conflated "tests aren't useful" with "*unit* tests aren't useful". I am asserting the latter, not the former! Testing in general is absolutely something that you can and should do when working on a production kernel

> The vast majority of production failures in kernel code are due to race conditions and other timing-specific issues

This assertion calls for links to data that backs this up I think.

Published counterarguments seemed to be easy to find, eg this survey suggests that most bugs are reproducible ("bohrbugs"): https://xiaotingdu.github.io/publications/TR2018.pdf

(journal link: https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=24)

I think that is fair to say for many hardware device drivers, but things like filesystems definitely should have a place for automated testing, eg. just having a bunch of golden values that verify correctness in backward and forwardcompatibility of the fs drivers should be great.

> but things like filesystems definitely should have a place for automated testing

And that is what xfstets is used for.

I don't suppose those long-running stress tests can be automated in any way, can they?

They are.

Many many companies run tests against the various kernel trees/branches, and will flood you with emails if you break the build.

I can't remember who (I think IBM maybe?), but Greg KH said one company made a fantastic test suite out of nowhere. If you submit six patches, it will try each one, tell you what you broke in patch 3, and recommend a way you can fix it. And this just gets emailed to the person who submitted the patches, and the maintainers if they actually merged it.

He said it was insanely advanced, and was made without the knowledge of the kernel maintainers. A company just made it and started sending emails one day.

Is there are literature on that IBM system? It sounds fascinating

Perhaps they’re talking about the (Intel) kbuild bot?

They are. There's a project by google called syzcaller, which pretty much does that (it's a bit more intelligent than that, and also includes running sanitizers, which find otherwise hard to detect memory safety and concurrency issues).

It's actually causing an unexpected problem: It finds so many issues that kernel devs have troule keeping up fixing them.

So it turns out kernel does have unit tests, just that they are running after release.

No, this is way above the unit test level. Fuzzers usually are.

Excellent! I keep complaining after each vuln pops up here that there isn't anyone with a kernel pipeline. Hopefully they have some static analysis too? Does the data out of this effort support the ggp that most of these defects are concurrency related vs the norm of memory safety?

Honestly, it's pretty rare I find a use case for testing period.

If I'm writing a library which contains algorithms or something like that it's pretty easy, seems useful or necessary.

But a lot of the time I'm gluing things together through API calls and it just seems pointless ?

It’s exactly the other way around for me. Because a test takes one input and checks the output.

An algorithm for, say, quick sort, will “test positive” on the same unit test that uses a brute force approach. A performance test might pick some differences up, but at that point that’s not a unit test any more

So basically, you find it important to test already tested code, with...your own tests?

Yes, that is precisely what I said, you got it right. Have a good day!

> The vast majority of production failures in kernel code are due to race conditions and other timing-specific issues, which unit tests are extremely poor at reproducing in a way that mimics how the code is used in real life.

Is this not due to the legacy of a language that allows race conditions? Take Loom [1] in Rust which is a tool to test your concurrency primitives, ensuring that deterministically all interleavings are tested and verified to be sound. Then, when you expose a safe interface, you know that the only potential issues you have are user logic errors or not covering the entire problem space with your tests.

https://docs.rs/loom/latest/loom/

You also interact with hardware. An RC-safe language does nothing when you suddenly realize there is an RC baked into the disk driver you are using and if you send it specific commands in a specific order too fast it does something unexpected.

Yes, that widens the problem space considerably, especially if it depends on the rate of access or something similarly non-deterministic. But you're still fundamentally testing interleavings through a synchronization library.

Still, reducing the problem space to such problems instead of any time you reach for atomics or a mutex, I imagine, removes the majority of bugs, so lets not throw the baby out with the bathwater because it's not perfect.

I'm surprised that I had to scroll this far down to find a Rust fanatic complaining about C.

I complain similarly about writing deeply multithreaded code with synchronized data structures in Java, Go or whatever. It's a nightmare the day you want to refactor anything and have to keep the entire implicit state in your mind to make sure you don't trip over yourself.

Race conditions are not limited to those “caused by language”, whatever that means.

Reply on news.ycombinator.com