Hacker News Reader

Comment by drewcrawford | Hacker News Reader

I need to ask a stupid question. What does one do with code coverage data?

Like, I know what it is. What I'm asking is, on what occasion would the presence of code coverage data be more useful than stepping through the code in a debugger. What is something you do differently having this data than not?

Like, is there a code review gate that forbids the code coverage figure to go below 80%? Or do developers consult the reports in a self-motivated way for some reason? Regularly? What is the step between "collect code coverage reports" and "profit"?

We recently got this feature in our compiler, and I guess it seems obvious to everyone why to use it, but I must have missed that day in class. So everybody's really excited to create the report, but I can't figure out what they do with it afterward that explains the whole exercise.

View on HN

Sometimes there's a disconnect between what you think you're testing and what you're actually testing.

After writing a unit test, I like to look at the code coverage analysis for that unit test and see if it actually tests the code that I think it tests. After seeing the analysis, sometimes I realize I'm not testing for a weird corner case that I should be.

It also slightly gamifies unit testing because there's something slightly satisfying in seeing 100% code coverage (though it's important to remember that 100% code coverage doesn't necessarily mean your tests are perfect or test all the conditions you should be).

We actually hook it into our build process. Basically:

* we have 100% code coverage except for some well-defined exclusions (e.g. // LCOV_EXCL_LINE comments to exclude code that is logically unhittable)

* on each checkin, our continuous integration server (Jenkins) runs the tests and checks that coverage is still 100%

* if not, it fails the build

This means that _every new checkin_ either has unit tests that exercise all the code, or // LCOV_EXCL comments that make it obvious to a code reviewer what isn't covered (and they can then sensibly judge the risk). It's a good way to keep our code quality high, and avoid the possibility that we quietly skimp on unit tests when under deadline pressure. Also, if new members of the team don't realise how important good unit test coverage is, they'll find that out on their first checkin when the build breaks, rather than several weeks in.

With all-due respect to other philosophies than mine, that sounds like a nightmare. And I like tests. But everything has a cost, and I can't imagine ever getting anything significant done with such a straight-jacket, I'd spend all day tracking down weird compiler quirks.

I do a lot of cross platform stuff, and just having warnings as errors (a good idea) means a lot of build fixing because clang and MSVC disagree on things. If I were writing code in MSVC and I had to deal with code coverage reports from clang breaking the build, I would be paralyzed.

"Code quality" isn't a thing that can be trivially measured, and in my experience people that fixate on certain metrics tend to miss the forest for the trees.

I can see that it wouldn't always be ideal, but on this (admittedly relatively niche) codebase - core telephone network infrastructure, which we only support on Ubuntu and which we build with GCC - the kind of clang/MSVC incompatibilities you talk about aren't a problem, and the reliability requirements are high enough that it's worth having CI check for 100% test coverage.

I think it's a question of field rather than philosophy - someone else in this thread said they do the same thing in avionics.

Claiming coverage has anything to do with reliability is just fooling yourself, at least if it's done on line or branch level. Will your 100% line coverage catch the divide by zero bug in this function?

    int foo(int x)
    {
        return 1 / (5 - x);
    }

One use case already pointed out is to check for gaps that your tests have not covered.

Other use case is figuring out what tests to re-execute when you change something - you can skip tests that never touch any code you are changing. Can be a time-saver if you have a big test suite or lack finer-grained tests (component/unit).

This sounds a bit dangerous. Code coverage works on code, not on data, so if you're testing something like:

    // globals
    rules_t rules[] = {{1, OK}, {2, FAIL}, {3, THERMONUCLEAR_WAR}};

It sounds like you could change rules 1 and 2 to THERMONUCLEAR_WAR as well and none of the tests would rerun. Is that right?

You might do something like this for local development, keeping things fast with some confidence you're not grossly breaking things. But you run the full suite through CI.

> is there a code review gate that forbids the code coverage figure to go below 80%?

That's one usage pattern.

Here are two others, more focused on individual productivity rather than team standards.

1. Write some code, write a test, run the test. Rather than stepping through the test with a debugger, you enable code coverage and can see the code path instantly. Same idea as a debugger, just faster and less precise. Occasionally a test passes for the wrong reason. Looking at coverage data is a fast way to find out that you didn't quite understand what your test did.

2. Write some code, and write a quickcheck-style test (generate random data, assert that properties hold on the output). Look at the code coverage, and see if the random data is covering all of the code paths. If it is, hurrah, you just built a great test with almost no effort. If the random data isn't triggering some code paths, you can now target your efforts on that unexplored branch.

Think of it this way. What are the odds of the code working correctly when the first time it is run is when it is in the customer's hands?

I've used code coverage analyzers on and off for 30 years. I know that merely executing code does not mean it is bug free. But I've found a very strong relationship between higher coverage and fewer bugs found in the field.

There've been reports on reddit that this is not so in other companies, but I get the feeling they're doing something wrong, because it utterly is not my experience.

I saw it used in the case of limited machine hours, if the test content is so huge, you can't run it all per checkin, you can use code coverage to get the minimum subset of tests that covers the maximum amount of code.

It's required in avionics software. Ideally 100% of the code in an avionics component will be executed in testing, but whatever isn't needs to be explained as to why it's acceptable to have been missed.

Code coverage is simple a measure of the amount of the code touched by your tests.

It doesn't tell you the quality of effectiveness of the tests. Only what code got executed during your tests.

Run your test suite, look at the code coverage report, find out which bits of your code doesn't get exercised by the test suite, become aware that changes in those bits of your code probably won't trigger any failing tests if they introduce breakage.

Another language. But I find this kind of usage pretty useful http://www.ncrunch.net

Basically instant feedback on performance and conformance.

In the newer version of intellij once you run code coverage it highlights in the side bar if that particular line was covered or not, its a pretty neat feature.

Reply on news.ycombinator.com