Code Coverage for Arbitrary Languages Made Easy [pdf] | Hacker News Reader

Code Coverage for Arbitrary Languages Made Easy [pdf]

semdesigns.com

23 pointsabricq20 comments

> Knowing that software is well tested is even more difficult. One widely accepted measure is that of branch coverage, the ratio of basic code blocks that were exercised by some test, to the total number of code blocks in the system under test

Widely accepted, but still surprisingly easy to game if you want to do it (and have lazy or no code review). Code coverage only tells you that some test has called the code, not that it has actually checked the results it gets. So, if you write a unit test that "exercises" all the various branches of a method and simply ignores (or performs only minimal checks on) the return values, you get 100% code coverage and, as a bonus, a test that's unlikely to ever break! This might still be useful to make sure that the code doesn't throw any exceptions etc., but it's contrary to the spirit of how unit tests should be written...

Mutation testing can help with that. With mutation testing the program is changed automatically in small ways to see if that makes a test fail.

https://en.wikipedia.org/wiki/Mutation_testing

That said, any metric is only useful for people actually using it to improve something, and completely useless when used in an adversarial way against someone willing to game the system. Coverage is pretty useful for a programmer to see where test gaps are, and easily gamed when it's felt to be too onerous.

> a test that's unlikely to ever break!

In a large code base I'm pretty sure you would end up throwing exceptions in lots of places.

Totally agree that code coverage is not a sufficient metric for quality, but it's still an interesting one.

At my company we have this language that is not mainstream (Oberon) and therefore there was no code-coverage reporting at all. I found this paper and realized it was super easy to write a minimal code-coverage analysis from our AST parser and code formatter (internal tools). Somehow it had never occured to me that it was this simple.

The transition from "no code coverage at all" to "seing which module is covered and which is not" was really insightful !

As someone that worked in projects with 100% guidelines for nice QA dashboards, eventually everyone games the system in large project, unit tests are written to cover all execution flows, not necessarly being actual tests.

Thus every code refactoring becomes a pain, as the tests aren't really testing the expectations of a function call, rather ensuring every line of code is shown as being covered.

  bool check_index_in_range(const std::vector<int>&v, size_t index) {
    // BUG ON NEXT LINE: SHOULD BE < NOT <=
    if(index <= v.size()) {
      // BRANCH A
      std::cout << "access was safe" << std::endl;
      return true;
    }
    else {
      // BRANCH B
      std::cout << "access was unsafe" << std::endl;
      return false;
    }
  }
  
  void main()
  {
    auto v = std::vector<int>();
    v.push_back(1);
    v.push_back(2);
    v.push_back(3);
  
    // Assertion passes, prints that the access is safe.
    // Exercises branch A
    assert(check_index_in_range(v,1));
  
    // Assertion passes, prints that the access is unsafe
    // Exercises branch B
    assert(!check_index_in_range(v,5));
  
    // We now have 100% code coverage
    // but still failed to catch a trivial bug.
    // Even though the buggy line was actually tested twice.
    // Code coverage is not a good way to measure testing
  }

Of course code coverage should not be seen as the only way to measure testing. But should it be systematically disregarded ? I rather see it as one of the tool for QAs to know where they stand at : 5% code coverage vs 85% is still an interesting information to have.

100% coverage just gives you a false sense of security. I'd rather just avoid the metric completely

Why does it give a false sense of security? It is not like >99% coverage is the finish line; it is the starting line, the bare minimum. That is like saying you get a false sense of security when a handgun does not pierce your newly developed tank armor.

Because being green on the QA dashboard doesn't mean the unit test makes any sense.

I aim for 100% not for the security but to know my code can be played with and poked at any level.

> One widely accepted measure is that of branch coverage, the ratio of basic code blocks that were exercised by some test, to the total number of code blocks in the system under test

I know that statement is true, but WHY is it true? I have never seen anyone look at a coverage report and take any useful action. Instead I see management looking at those reports for a number to rate people on, but no useful action is ever taken. When I've looked at the reports I keep finding trivial code that would be hard to test for little gain.

I eliminated the report and nobody misses it. Instead we report to management total number of tests run (test cases, and each assert). Which is easy to game, but since the number naturally increases fast isn't worth the bother.

It is no different than any other hard measurement. What is the best heart rate for you, as an example?

In many, it is more a directional check. 100% is likely not to give you 100% confidence. But, 0% will make everyone less confident. And if you have two teams, one at 50% and one at 80%, it is probably safe to assume that the testing culture is better for the 80% team.

And why branch coverage, as opposed to path coverage? Largely because you hit the easy to measure things before you worry about the hard to measure ones.

I know that statement is true, but WHY is it true? I have never seen anyone look at a coverage report and take any useful action.

We made code coverage a part of CI, and we've been using that to drive up code coverage. It's been working. Our tests have gone from ~25% to ~50% coverage. Breakages making it out to staging have gone down.

Also, some incentives are well aligned with this. If people see that things aren't easy to test, then they get rearranged to make them easy to test. Not all incentives are aligned, but not everything is perfect. This is why reviews are useful.

When I've looked at the reports I keep finding trivial code that would be hard to test for little gain.

If your trivial code is also hard to test, then this indicates architectural code debt. Why isn't the trivial code trivial to test?

How do you test the code that calls a OS function? (think of something like shutdown - you don't want your test to turn off your CI system) The obvious answer is write a wrapper and a mock implementation of the wrapper. Now you have the wrapper that is untested, so you write a wrapper for that so you - I actually had someone go down this 3 levels before giving up.

I work in embedded where we have to control real hardware.

The other thing you can do is composition of individually-verified parts. In your example, you make tests that really exercise the shutdown function. Also, note any preconditions these functions need to meet so the application calling it can be consistent with specs. From there, just assume shutdown works if used within the tested spec.

This method also encourages reusing working specs, code, and tests. We see that in DO-178C with the RTOS’s, etc.

Thank you for posting! We have a rust application that we compile to wasm and run. (there's no way to run it by compiling to native code.) We have tests that compile our code to wasm and run them in our execution environment. We want to get code coverage set up for our tests, but we had no idea how to do that in our environment.

This article suggests a way to have code coverage probes automatically added to various locations in our source. I think I would also need some way to read coverage information from the rust program itself, which seems straightforward. But I wonder if anyone knows if this has already been done? I know about the llvm-based coverage reports, but I don't know if they can easily be adapted to this purpose

I like the term "industrial-strength" transformation system. That's what I'm building, as it happens. I wonder if the existence of a completely free one might not be so good for this company...

Oh my, their pricing undercuts the paper's argument a bit: http://semdesigns.com/Purchase/

You have to buy each piece of functionality they offer separately for each langauge, for example I can buy a Java test coverage tool for $200, and it'll still only run on Windows or in Wine.

related to the "This technology has been used to construct production test coverage tools for ... PARLANSE (a parallel language used to implement DMS itself)" part:

https://en.wikipedia.org/wiki/DMS_Software_Reengineering_Too...

https://www.semanticdesigns.com/Products/Parlanse/examples.h...

Reply on news.ycombinator.com