Clang vs. GCC – code coverage checking | Hacker News Reader

Clang vs. GCC – code coverage checking

rkd.me.uk

98 pointsrkday35 comments

Am I reading this right? He says he is comparing clang 3.8, which is a month old, to gcc 4.8, which branched over 3 years ago. I would rather see what gcc 6 does with this. That release should happen in a month or so.

Very good point. I missed that.

I wouldn't read too much into the article. It's just a blog post of a dude reporting what he saw when he recompiled his code with a different compiler.

Aye, but it shows something important. Testing your work with other toolchains and toolings can help discover weaknesses not only in your code, but in the rest of your tooling. Strength in numbers is useful.

The only gcov-related change listed in the 4.9, 5 and 6 release notes is the addition of a new tool, so I wouldn't expect there to be meaningful changes.

Regardless, I think the author's conclusion is overly specific and really should have just been "trying all of the various tools for doing something is useful even if they're theoretically comparable".

Thanks for pointing out how old GCC 4.8 is! You're right that a comparison against a later version of GCC is fairer - I've tested with GCC 5.3 (the latest I can easily get) and updated the blog post. (5.3 doesn't spot the missing coverage either.)

That is great, thanks. It just seemed odd to compare compilers of such different age. I have nothing against clang, it certainly is faster in most cases.

As others said, it makes sense for everyone to try different toolchains and see what works. It is possible to test newer gcc versions by installing in /opt or something, just make sure to get the right libstdc++ at runtime. With ABI changes, updating the distro gcc is too much trouble.

I need to ask a stupid question. What does one do with code coverage data?

Like, I know what it is. What I'm asking is, on what occasion would the presence of code coverage data be more useful than stepping through the code in a debugger. What is something you do differently having this data than not?

Like, is there a code review gate that forbids the code coverage figure to go below 80%? Or do developers consult the reports in a self-motivated way for some reason? Regularly? What is the step between "collect code coverage reports" and "profit"?

We recently got this feature in our compiler, and I guess it seems obvious to everyone why to use it, but I must have missed that day in class. So everybody's really excited to create the report, but I can't figure out what they do with it afterward that explains the whole exercise.

Run your test suite, look at the code coverage report, find out which bits of your code doesn't get exercised by the test suite, become aware that changes in those bits of your code probably won't trigger any failing tests if they introduce breakage.

Sometimes there's a disconnect between what you think you're testing and what you're actually testing.

After writing a unit test, I like to look at the code coverage analysis for that unit test and see if it actually tests the code that I think it tests. After seeing the analysis, sometimes I realize I'm not testing for a weird corner case that I should be.

It also slightly gamifies unit testing because there's something slightly satisfying in seeing 100% code coverage (though it's important to remember that 100% code coverage doesn't necessarily mean your tests are perfect or test all the conditions you should be).

Code coverage is simple a measure of the amount of the code touched by your tests.

It doesn't tell you the quality of effectiveness of the tests. Only what code got executed during your tests.

> is there a code review gate that forbids the code coverage figure to go below 80%?

That's one usage pattern.

Here are two others, more focused on individual productivity rather than team standards.

1. Write some code, write a test, run the test. Rather than stepping through the test with a debugger, you enable code coverage and can see the code path instantly. Same idea as a debugger, just faster and less precise. Occasionally a test passes for the wrong reason. Looking at coverage data is a fast way to find out that you didn't quite understand what your test did.

2. Write some code, and write a quickcheck-style test (generate random data, assert that properties hold on the output). Look at the code coverage, and see if the random data is covering all of the code paths. If it is, hurrah, you just built a great test with almost no effort. If the random data isn't triggering some code paths, you can now target your efforts on that unexplored branch.

We actually hook it into our build process. Basically:

* we have 100% code coverage except for some well-defined exclusions (e.g. // LCOV_EXCL_LINE comments to exclude code that is logically unhittable)

* on each checkin, our continuous integration server (Jenkins) runs the tests and checks that coverage is still 100%

* if not, it fails the build

This means that _every new checkin_ either has unit tests that exercise all the code, or // LCOV_EXCL comments that make it obvious to a code reviewer what isn't covered (and they can then sensibly judge the risk). It's a good way to keep our code quality high, and avoid the possibility that we quietly skimp on unit tests when under deadline pressure. Also, if new members of the team don't realise how important good unit test coverage is, they'll find that out on their first checkin when the build breaks, rather than several weeks in.

With all-due respect to other philosophies than mine, that sounds like a nightmare. And I like tests. But everything has a cost, and I can't imagine ever getting anything significant done with such a straight-jacket, I'd spend all day tracking down weird compiler quirks.

I do a lot of cross platform stuff, and just having warnings as errors (a good idea) means a lot of build fixing because clang and MSVC disagree on things. If I were writing code in MSVC and I had to deal with code coverage reports from clang breaking the build, I would be paralyzed.

"Code quality" isn't a thing that can be trivially measured, and in my experience people that fixate on certain metrics tend to miss the forest for the trees.

I can see that it wouldn't always be ideal, but on this (admittedly relatively niche) codebase - core telephone network infrastructure, which we only support on Ubuntu and which we build with GCC - the kind of clang/MSVC incompatibilities you talk about aren't a problem, and the reliability requirements are high enough that it's worth having CI check for 100% test coverage.

I think it's a question of field rather than philosophy - someone else in this thread said they do the same thing in avionics.

Claiming coverage has anything to do with reliability is just fooling yourself, at least if it's done on line or branch level. Will your 100% line coverage catch the divide by zero bug in this function?

    int foo(int x)
    {
        return 1 / (5 - x);
    }

Another language. But I find this kind of usage pretty useful http://www.ncrunch.net

Basically instant feedback on performance and conformance.

I saw it used in the case of limited machine hours, if the test content is so huge, you can't run it all per checkin, you can use code coverage to get the minimum subset of tests that covers the maximum amount of code.

One use case already pointed out is to check for gaps that your tests have not covered.

Other use case is figuring out what tests to re-execute when you change something - you can skip tests that never touch any code you are changing. Can be a time-saver if you have a big test suite or lack finer-grained tests (component/unit).

This sounds a bit dangerous. Code coverage works on code, not on data, so if you're testing something like:

    // globals
    rules_t rules[] = {{1, OK}, {2, FAIL}, {3, THERMONUCLEAR_WAR}};

It sounds like you could change rules 1 and 2 to THERMONUCLEAR_WAR as well and none of the tests would rerun. Is that right?

You might do something like this for local development, keeping things fast with some confidence you're not grossly breaking things. But you run the full suite through CI.

Think of it this way. What are the odds of the code working correctly when the first time it is run is when it is in the customer's hands?

I've used code coverage analyzers on and off for 30 years. I know that merely executing code does not mean it is bug free. But I've found a very strong relationship between higher coverage and fewer bugs found in the field.

There've been reports on reddit that this is not so in other companies, but I get the feeling they're doing something wrong, because it utterly is not my experience.

It's required in avionics software. Ideally 100% of the code in an avionics component will be executed in testing, but whatever isn't needs to be explained as to why it's acceptable to have been missed.

In the newer version of intellij once you run code coverage it highlights in the side bar if that particular line was covered or not, its a pretty neat feature.

"Build your code with more than one compiler" was good advice 20 years ago, and is still good advice today :-)

Dear God, now I remember why I do not program in C++ anymore (Go and C# all the way!). That pre-11 code looked awful. If networking was nicer, I might actually start considering recoding some of my projects in C++.

It's hard to say what he was trying to accomplish, but he probably could have written something like this.

    auto index = std::distance (begin(replicas),
                   std::find(begin(replicas), end(replicas), my_ip));
    return (replicas.size() == index ? -1 : index);

That's a bit harder to read isn't it? I think finding the location in first line and using distance in the ternary operator makes it more readable.

Like this?

    auto it = std::find(begin(replicas), end(replicas), my_ip);
    return (end(replicas) == it ? -1 : std::distance(begin(replicas), it));

Yeah, i'll go with that.

Yeah that's what I was thinking.

If you have £40 or so, and a few evenings free, get the C++11 version of the Stroustrup programming language book and see what's changed. Modern C++ looks entirely different to older C++. Templates still a bit scary, so learn them or ignore them.

Yes, but are there still tons of boilerplate? Or is there a 11/14/17 library that incredibly decreases the amount of code needed? For example a HTTP listener with routing?

A good way to represent a branch coverage miss.

I love stuff like this. Short, understandable.

please look http://www.expcov.com

Reply on news.ycombinator.com