Hacker News Reader

Comment by rob74 | Hacker News Reader

> Knowing that software is well tested is even more difficult. One widely accepted measure is that of branch coverage, the ratio of basic code blocks that were exercised by some test, to the total number of code blocks in the system under test

Widely accepted, but still surprisingly easy to game if you want to do it (and have lazy or no code review). Code coverage only tells you that some test has called the code, not that it has actually checked the results it gets. So, if you write a unit test that "exercises" all the various branches of a method and simply ignores (or performs only minimal checks on) the return values, you get 100% code coverage and, as a bonus, a test that's unlikely to ever break! This might still be useful to make sure that the code doesn't throw any exceptions etc., but it's contrary to the spirit of how unit tests should be written...

View on HN

Mutation testing can help with that. With mutation testing the program is changed automatically in small ways to see if that makes a test fail.

https://en.wikipedia.org/wiki/Mutation_testing

That said, any metric is only useful for people actually using it to improve something, and completely useless when used in an adversarial way against someone willing to game the system. Coverage is pretty useful for a programmer to see where test gaps are, and easily gamed when it's felt to be too onerous.

Totally agree that code coverage is not a sufficient metric for quality, but it's still an interesting one.

At my company we have this language that is not mainstream (Oberon) and therefore there was no code-coverage reporting at all. I found this paper and realized it was super easy to write a minimal code-coverage analysis from our AST parser and code formatter (internal tools). Somehow it had never occured to me that it was this simple.

The transition from "no code coverage at all" to "seing which module is covered and which is not" was really insightful !

As someone that worked in projects with 100% guidelines for nice QA dashboards, eventually everyone games the system in large project, unit tests are written to cover all execution flows, not necessarly being actual tests.

Thus every code refactoring becomes a pain, as the tests aren't really testing the expectations of a function call, rather ensuring every line of code is shown as being covered.

  bool check_index_in_range(const std::vector<int>&v, size_t index) {
    // BUG ON NEXT LINE: SHOULD BE < NOT <=
    if(index <= v.size()) {
      // BRANCH A
      std::cout << "access was safe" << std::endl;
      return true;
    }
    else {
      // BRANCH B
      std::cout << "access was unsafe" << std::endl;
      return false;
    }
  }
  
  void main()
  {
    auto v = std::vector<int>();
    v.push_back(1);
    v.push_back(2);
    v.push_back(3);
  
    // Assertion passes, prints that the access is safe.
    // Exercises branch A
    assert(check_index_in_range(v,1));
  
    // Assertion passes, prints that the access is unsafe
    // Exercises branch B
    assert(!check_index_in_range(v,5));
  
    // We now have 100% code coverage
    // but still failed to catch a trivial bug.
    // Even though the buggy line was actually tested twice.
    // Code coverage is not a good way to measure testing
  }

Of course code coverage should not be seen as the only way to measure testing. But should it be systematically disregarded ? I rather see it as one of the tool for QAs to know where they stand at : 5% code coverage vs 85% is still an interesting information to have.

100% coverage just gives you a false sense of security. I'd rather just avoid the metric completely

Why does it give a false sense of security? It is not like >99% coverage is the finish line; it is the starting line, the bare minimum. That is like saying you get a false sense of security when a handgun does not pierce your newly developed tank armor.

Because being green on the QA dashboard doesn't mean the unit test makes any sense.

I aim for 100% not for the security but to know my code can be played with and poked at any level.

> a test that's unlikely to ever break!

In a large code base I'm pretty sure you would end up throwing exceptions in lots of places.

Reply on news.ycombinator.com