Hacker News Reader

Comment by latch | Hacker News Reader

code coverage numbers are ridiculously stupid.

I know you aren't attacking me. I realize that for this simple method, the code coverage is, well, very good (perhaps to a point of being bad). This is an contrived example to showcase the implication of using mocks versus stubs though....or more abstractly to show the difference, and the different way to test, behaviors vs implementation logic. Whether these always have to be tested, I agree is worthwhile to talk about.

I don't wanna become the "Programming, Motherfucker" example of what's wrong with software development :)

View on HN

There is only one good code coverage number and it's "100". If you aim for 100% code coverage then it really helps make your code better as you must test all the parts, not only the easy ones - that forces you to write code that's easier to test, and "code easier to test" equals "better code" (more sane interfaces/API, decoupled classes, etc. etc.).

However, you shouldn't fall into the fallacy that if you have 100% "code coverage" that you code is really fully tested.

I disagree. Maybe flamebait, but I think 100% is usually either a silly goal, or a good start. 100% for a complex project is basically impossible to achieve (and assumes every area of the code is as important as the next), and for a lot of code 100% is nothing. Consider a function that divides to arguments and returns the result. I can pass in div(2,2). There, done. 100% coverage.

[edit] I forgot to mention what value I do think it has. On teams that I'm on I always say I frankly don't care what the number is, as long as it's going up and not down.

Of course 100% code coverage won't make your code better in every possible aspect, nor it guarantee that tests are perfectly written, but at least a) the tough parts of the code won't be skipped during the test b) you will have cleaner, more usable (testable) API.

I still think it's a bit of a wonked metric. The goal out to be to increase your coverage of possible test cases, not LOC. Since you can't cover every possible test case on any non-trivial project, they must have priority. Priority is probably something like how likely the case is to occur, balanced against the consequences of a failure.

Test coverage is a simple indicator, and simple indicators are handy.. but chasing 100% too blindly will cause you to start seeing every line of code as equal, and they're not.

[edit] - oh, I meant to mention that both of your points (a & b) make sense. I can see how chasing coverage could help make sure you test parts of your code you've been avoiding because they're hard to test. I just wonder if it's the best way to accomplish that goal.

I've seen "100% coverage" specs that managed to run every line of code while only testing trivial, obviously correct, aspects and failing to test any of the actually important behavior. This sort of stuff gives TDD a bad name, but is, in my experience, sadly common.

Also, at least in the Ruby community, people seem to only measure line coverage. Without branch and conditional coverage reports, it's possible to run 100% of the lines but still miss a lot of code paths.

I'll tell you what we do in hardware to get test coverage that is worth way more than 100%: we do fault injection. I have never heard of anyone trying to do the same in software, but it would be interesting.

Here's how it works:

for each statement in your code, break it, then run your entire test suite. If none of your tests have uncovered a problem, mark the code as uncovered. Repeat for each code statement.

It's crazily expensive to compute, but it's so much more accurate than to say that a line of code was hit or not. It measures whether the code had a functional impact on the overall behavior.

That's generally referred to as 'mutation testing'. There are a number of systems that enable this in dynamic languages, the one I am most familiar with is Heckle: https://github.com/ryansobol/heckle

Mutation testing is great, it highlights all the areas where your tests aren't written well, and teaches you to write better tests. I'm not sure that using it all the time is the best use of resources in most cases though...

Reply on news.ycombinator.com