Hacker News Reader

> Well you executed the branch/line at least once with one potential input. Was it an edge case input or a happy path input?

How does that matter? If something about the input causes a difference in the execution of the code, then 100% coverage means you necessarily tests both kinds of input. You can't reach the edge case branch with the happy path input.

Now, if your code is just pumping data from one point to another point, it's possible that the destination will vary its behavior based on the input in a way that your coverage-based testing can't see. But if you're doing something with the input yourself, then 100% coverage means you tested every possible kind of input.

(Actually, a "branching" problem can still arise if you have something like

  if( condition1 ) {
    do_stuff();
  }
  
  if( condition2 ) {
    do_other_stuff();
  }
  
  if( condition3 ) {
    do_other_other_stuff();
  }

Because with this code, it's possible to get test coverage of every line of code (there are 6 of them, not counting the '}'s) while not covering every branch (there are 8 of them, depending on the combined truth values of condition1, condition2, and condition3).)

That's because you have such simple conditions. How do you compute them?

   if (x < y) { ... }

really changes things. Did you test for x === y? for NaN? for very close doubles that should have passed/failed for business reasons? For types other than numbers if your language allows?

Expressions can have any number of edge cases that code coverage can't account for.

Well, imagine two different analyses of the code:

1. Does each branch do the right thing?

2. Does the code go down the correct branch in the first place?

I've focused more on the first question, and you're focusing more on the second. But it's fair to say that each question is a reasonable focus of testing, which weakens my comment above.

Depends on the language/type safety offered. If your language has a NaN (javascript) and you can assert the expected behavior of x < NaN (false) or if you don't even want to encounter that, then you're able to reason back to the generation of the inputs. Wherever x and y came from, if there was a possibility of an NaN being a value, you want the generating function to exception or cast via some sort of validation before handing them off to other unsuspecting functions.

In my experience, when a function has more than 1 if statement, it's probably hard to test and you might want to split it up.

When a function has has more than 2 if statements, you definitely want to break it up.

Imagine the function where conditions aren't related at all, as you posited:

    myfn(int x, int y, string foo) {
        if(foo == "bar") {
            do_stuff();
        }
  
        if( x < y ) {
            do_other_stuff();
        }
  
        if(time() == "noon") {
            do_other_other_stuff();
        }
    }

What's really happening here? Why is all this wrapped in 1 function, when the args aren't related at all, nor the work they are dependent on?

Let's say the caller was doing: myfn(1,2,foo);

I would split this function into 3 different function calls in the caller...

    possiblyDoStuff(foo);
    possiblyDoOtherStuff(1,2);
    possiblyDoOtherOtherStuff(time());

Let's use something that you wouldn't simply refactor upstream:

    myfn(string go, string for, string foo) {
        if(go == foo) {
            do_stuff();
        }
  
        if(for == go) {
            do_other_stuff();
        }
  
        if(foo == for) {
            do_other_other_stuff();
        }

        someOtherFn(go, for, foo);
    }

Move that complexity into smaller chunks:

    myfn(string go, string for, string foo) {
        possiblyDoStuff(go, foo);
        possiblyDoOtherStuff(go, for);
        possiblyDoOtherOtherStuff(for, foo);
        someOtherFn(go, for, foo);
    }

Now you have the 7 tests. 2 for each possibly = 6 which are pretty easy and 1 for myfn. It is exceedingly rare that functions require this kind of attention. There are usually a bunch of side effects or return values that are dependent on these checks. eg:

    myfn(string go, string for, string foo) {
        // examples of why you are calling these, to get vals
        var go2 = possiblyDoStuff(go, foo);
        var for2 = possiblyDoOtherStuff(go, for);
        var foo2 = possiblyDoOtherOtherStuff(for, foo);
        return { go2, for2, foo2 };

        //move this to caller someOtherFn(myfn(go, for, foo));
    }

This isn't true at all. Consider buffer overflows, bugs in your bitwise arithmetic, division by zero, or null pointer exceptions.

> Is it just me or was this _not_ surprising at all?

It wasn't surprising to anyone that has reflected about the value of tests. Mindless testing/TDD isn't usually reflective, though.

It’s hard to do test-first TDD “mindlessly” because writing a test usually forces you to think in terms of the specification of the behavior you’re about to implement.

> It’s hard to do test-first TDD “mindlessly” because writing a test usually forces you to think in terms of the specification of the behavior you’re about to implement.

Tests are code. Code can be sloppy, fallible, useless. Writing a consumer before you write a provider doesn't make either more robust, just the point at which they meet more clear. You can certainly write a test that uses a function, but the test doesn't actually test anything at all, just that the function exists. aka "mindless TDD".

Very well said.

I have nothing against TDD if that's something that helps a particular person write good code and good tests.

I find though that people that write good tests are just people that write good tests. Most people write tests that assert on implementation details instead of inputs and output and that's easily doable via TDD as well.

It’s hard to assert based on implementation details if you don’t have an implementation yet: the advantage of test-first code is that you usually have to write the tests in terms of the interfaces of the input and output types rather than inspecting the code under test.

That's because you know how to write good tests but TDD didn't force you to do that.

Simply put, TDD just means you write the assertion first, see it red, then change the code to make it green.

You can do that by specifying a failing test via input/output testing. You can also very easily write a failing test that tests that the YXZ() method is gonna be called on some internal thing that you're replacing with a mock. Add that method and voila, green test, TDD happy. Bad test though.

Yes, I agree with this. Process exists to increase the bottom, but if you're trying to rely on TDD to get valuable tests, you're probably just thinking of the problem incorrectly (I think).

In my experience, the absolute best way to get valuable tests is to have someone whose job it is to own them, their correctness, and add to them when new issues arise. An SDET, or a rotating team member, or whatever. The average dev isn't going to write a test that's super valuable if they also wrote the code.

Like most others here in the comments, I find the results not particularly controversial. Pretty easy to imagine scenarios where lines of code are run, but the tests themselves are suboptimal or incomplete.

As an aside— I've often mused at to whether theres a more useful multivariate (but still coarse) measure that could combine coverage with cyclomatic complexity and number of tests/assertions. Seems like it would be incrementally better than just coverage alone.

If (as the study says) # of tests is the true correlate then

1. We need some way to compare # of tests across differently-sized codebases, i.e a percentage metric like coverage 2. The fact that correlation disappeared when controlling for absolute # of tests seems to suggest it's still encouraging people to write more tests. Once your org gets to a certain size you need these blunt instruments like minimum coverage requirements because you can't just rely on individual excellence anymore.

100% or you suck. You need 100% and then some for FULL coverage.

I work with people who turn in 80% PRs...these people are bug factories.