Coverage is not strongly correlated with test suite effectiveness | Hacker News Reader

Coverage is not strongly correlated with test suite effectiveness

neverworkintheory.org

242 pointszdw173 comments

It is easy to write a test that executes code without actually testing anything. I use coverage to find code with no tests all at, and write tests for that code. But once it is "covered" the coverage report is useless. In interpreted languages (ruby/python/etc) coverage at least tells you if there's a syntax error before running it in production, which is useful.

Test first also improves the quality of the tests just because it forces you to write a failing test first, and then write the code to make it pass, and you do this over and over as you build up the production code. In this way you are actually proving that the code you are adding is making tests pass.

That doesn't mean your tests are great - you are probably still mostly testing the happy path. But then you have a solid foundation to layer mutation/fuzz testing on.

I can't tell you how often I see people write tests after writing the production code - and I can go and delete most of the production code and the tests still pass. Writing quality tests for existing code is much harder than writing tests for code that doesn't exist - it is counterintuitive if you haven't done it both ways.

I was going to be worried if mutation testing wasn't mentioned here. Is a great way to test the effectiveness of your tests at catching the common mistakes people make in code.

That is, a mutation suit doesn't test your code, per se. It tests your test suit.

I've worked a lot on Java, and while there is great mutation testing tools (well, PIT specifically), I find them hard to "scale" practically, i.e. to run them in an automated fashion "every time". You either get long running builds with them in it, or you have to deal with the logistics of moving caches around (so that the mutation testing can be incremental). And as just another tool that a developer MAY use if they so choose to, it's... well... nice... but no longer front of mind, for most people on the team. I'm very interested if you're experiences have been different?

No, I found the same. Team disabled them on every release build, as they felt it was too slow. I may bring them back in the release servers, but it is honestly trying to argue for these when nobody else cares.

It needs someone who cares which us a thankless job but it’s useful as smote tests run nightly / weekly.

It tends to be better for broad concerns which trigger somewhat rarely, tend not to be otherwise caught, but can be quite concerning. Crater is a good example of that, it’s infeasible to build & test every public crate on every CI check, but it’s a useful sanity check when release arrives or when messing with the more subtle aspects of the compiler.

> But once it is "covered" the coverage report is useless.

Well, yes, any report that is guaranteed to say the same thing every time is useless.

I think it was Brian Marick who pointed out that the great benefit of a coverage report is that it tells you what you forgot to think about when you wrote the test suite. One response to code that isn't covered is to write the tests to exercise that code, but there are a couple of other possibilities he suggests might be better:

1. Can the uncovered code be removed from the system entirely? Maybe if none of the tests for other parts of the system invoked it, it's not used at all. (This is less likely if you use a lot of test stubs.)

2. Maybe if you didn't think about the scenario where the uncovered code gets invoked when you were writing the test suite, there are other things you also didn't think about—and maybe some of them aren't covered in the implementation either. Write down the missed case and put the implementation away for a while—hopefully long enough to forget how it's implemented. Once you've forgotten, refer to your notes and write a suite of tests that covers the missed case as well as anything similar.

— ⁂ —

I agree that test-first improves your tests in the way you describe: your test suite is guaranteed to have nearly complete coverage. Also, you have some evidence that the test itself works rather than vacuously passing.

But I think there are two other benefits of test-first programming that are commonly undersold.

First, it makes programming more fun, because you have immediate feedback when you make a test pass.

Second, sometimes you write a test for code you haven't written yet, and you can see by looking at the test that your design sucks: you need seven objects and six method calls to do something simple, and one of the method calls has a boolean parameter, making the code incomprehensible. This feedback allows you to improve your design, possibly several times, before writing the implementation. This allows for faster design iteration than refactoring the implementation toward a better design does. I think this is what jeffbee is saying in https://news.ycombinator.com/item?id=28677978.

I find test-first most useful when debugging a hairy problem, and I think that's somewhere it's not used enough.

Reproducing the bug consistently* is the first step to understanding how it works, and how to fix it. And then you get the thrill of trying to turn that test green as you tinker.

...

* Okay, sometimes "consistently" is "fails about 1/100 executions", but that's not so bad if you can run your unit test 1000 times in the span of 30s.

Oh, yeah, that's a good point; even people who don't do test-first most of the time will often do it for bug fixes.

I think the importance of performance to software quality via automated testing is profoundly underestimated.

> Writing quality tests for existing code is much harder than writing tests for code that doesn't exist

I’ve been trying this and the biggest place it’s helped me is parsing. Working out all the bad formats, types, unexpected stuff before I write the code was worth it.

It’s still probably low quality compared to what it will eventually be, but a pretty good way to start.

My favorite peer review trick is reverting the code change and showing that the test still passes. No tests will cover everything but its not hard to apply some simple tricks that raise the bar of actual coverage.

I found TDD useful for a well defined problem or an agreed-upon API. For apps for example, especially those not well defined and designed as-you-go, where the designer and PM might change their minds frequently after toying around with the app or getting user feedback, TDD is a lot of overhead and tests after writing the code are primarily useful for preventing regressions when somebody else changes your code.

I hear this a lot, but the result is usually an untested, and usually nearly untestable (because it was written without tests), prototype with a few characterization tests that gets pushed to production and haunts you for the rest of the life of the product.

Pototyping to define the problem or API is fine, but most people don't have the discipline to tear it out and start over when they finally do have a well defined problem.

It is somewhat unfathomable to me that you can have an idea if what production code to write, but no idea what test to write. Certainly you can "expect this page has a button" if you are about to add a button and write the test first. Or "expect this method to add a record to the database", etc. Certainly you are about to write code that does something, so just write a test to expect that thing to happen.

Especially in the case where someone is changing their mind all the time, you are dead in the water without tests to support your changes. You need to manually re-test everything every time they change their mind.

Testing after the fact is not that useful for catching regressions because it is unlikely enough tests will be written due to pressure to release. Also a much greater percentage of the tests written after the production code tend to be vacuous tests.

> the result is usually an untested, and usually nearly untestable

No way... if you ever wrote any tests, you can easily know how to write code that will be testable even if you do it later. Doing it before is just going to be a big waste of time if you throw the code away later, which happens a lot when you need to experiment with things before actually choosing what works best. Yes, you can do that with TDD as well but it will absolutely slow you down. If you end up not writing tests later, when the implementation has been chosen, it's just because you don't care about the code quality... I find it hard to believe TDD will fix you in that case.

I've always experienced that writing the actual code is but a minor part of building features (or fixing bugs). Especially when prototyping, which most often is merely ducttaping libs together.

Everyone remembers a bugfix of one line, that took hours, or days to find.

I'd stringly encourage you to check your commitlogs. You'll probably find you commit at most hundreds of lines a day, quite probably, as is my case too, av era aging under ten lines a day.

Please stop thinking that writing code which does not make it into commits, is waste. It is learning. And tests are by far the best place to learn about your, and Libs' code, APIs and behaviour.

Swings and roundabouts: what you lose on writing tests you gain on not bothering to implement things you don't need. If the test passes, you stop.

Without the tests to guide you it's very easy to waste time over-engineering, even if what you're building is well-structured.

I often write code without implementing anything, just the "surface API". That's when I find whether things will work or not. Tests are a hindrance to that. Once I figure out the design, then I will test all that I think is important.

> what you lose on writing tests you gain on not bothering to implement things you don't need.

I am not sure where to start... I've written so much code, applications, libraries, algorithms... and I am pretty sure the opposite of what you say is true: with TDD, I would've spent hours trying to get something working that later I would find, by exploration, that I didn't need at all.

> I often write code without implementing anything, just the "surface API". That's when I find whether things will work or not.

There's an entire school of TDD which works this way. That's not incompatible at all.

> with TDD, I would've spent hours trying to get something working that later I would find, by exploration, that I didn't need at all.

If you're writing tests for something you don't need, the problem isn't with TDD as I understand it.

This is my experience too (business/customer facing CRUD systems). Perhaps we don't spend long enough thinking about the problem before we start or planning enough, but most software I write starts with a business problem that needs a solution and data, inputs and processes will evolve as it's written. Normally because what comes out in business analysis and stakeholder interviews is close to but not quite what is wanted, but also because of user feedback from user testing (sometimes of a mockup, sometimes prototype software) which is so valuable but now the inputs change again.

I enjoy writing tests when developing libraries where the inputs and outputs are well defined. It's relaxing to do this kind of programming.

You don't need precise specs to practice TDD. If you have idea of the code you need to write, you can write the test for it beforehand - no matter how often someone might change their mind about the app.

Doing so would actually make your life a lot easier when it's time to alter functionality, because now you have well tested and testable code. Code that is written to be tested is usually a lot easier to reason about, to change and extend.

If you do it in concise manner and test behavior rather than implementation, what you previously thought of overhead will actually speed you up.

Tests are also meant to be continously refactored, though.

For prototyping TDD may be waste. If you are just trying different ux and different logic it may not help to test. For almost everything else I think it helps.

    In other words, more tests do find more bugs, but it's the number of tests and not their code coverage that has most of the predictive value. It's a surprising result, so if you'll excuse me, I have a couple of lecture slides on software testing I need to revise

Is it just me or was this _not_ surprising at all?

I mean I suppose I should have expected what he said, given it sometimes seems hard to convince other people about this but to me it's a well known fact. There are so many ways this can go wrong.

I mean it's so easy to give the one counter example needed to break the myth of 100% test coverage being good for much: Well you executed the branch/line at least once with one potential input. Was it an edge case input or a happy path input?

More tests than is needed for "100% coverage" means that you actually executed some lines multiple times, hopefully with not just 10 happy path scenarios but with 1 happy path and 9 edge cases. Now remove the happy path scenarios for trivial code and also the edge case scenarios for trivial code and your coverage might only be 80% but you have the same actual test suite effectiveness. When you keep adding tests, add more to the 9 edge cases, staying with the same coverage but make the suite more robust.

Of course 20% is better than 0% and 50% is better than 20%. Somewhere between 50 and 100 is an optimal point. For lack of a good estimate, let's go with the old 80/20 rule. Something around 80% test coverage is what you can use to make a tool fail the build. If you pass the 80% mark you _might_ have a good test suite but it doesn't mean you stop there. It's just a reminder that you might have missed adding tests when that thing triggers a red build but don't use it to mean that you actually have a good test suite.

> Is it just me or was this _not_ surprising at all?

It wasn't surprising to anyone that has reflected about the value of tests. Mindless testing/TDD isn't usually reflective, though.

It’s hard to do test-first TDD “mindlessly” because writing a test usually forces you to think in terms of the specification of the behavior you’re about to implement.

> It’s hard to do test-first TDD “mindlessly” because writing a test usually forces you to think in terms of the specification of the behavior you’re about to implement.

Tests are code. Code can be sloppy, fallible, useless. Writing a consumer before you write a provider doesn't make either more robust, just the point at which they meet more clear. You can certainly write a test that uses a function, but the test doesn't actually test anything at all, just that the function exists. aka "mindless TDD".

Very well said.

I have nothing against TDD if that's something that helps a particular person write good code and good tests.

I find though that people that write good tests are just people that write good tests. Most people write tests that assert on implementation details instead of inputs and output and that's easily doable via TDD as well.

It’s hard to assert based on implementation details if you don’t have an implementation yet: the advantage of test-first code is that you usually have to write the tests in terms of the interfaces of the input and output types rather than inspecting the code under test.

That's because you know how to write good tests but TDD didn't force you to do that.

Simply put, TDD just means you write the assertion first, see it red, then change the code to make it green.

You can do that by specifying a failing test via input/output testing. You can also very easily write a failing test that tests that the YXZ() method is gonna be called on some internal thing that you're replacing with a mock. Add that method and voila, green test, TDD happy. Bad test though.

Yes, I agree with this. Process exists to increase the bottom, but if you're trying to rely on TDD to get valuable tests, you're probably just thinking of the problem incorrectly (I think).

In my experience, the absolute best way to get valuable tests is to have someone whose job it is to own them, their correctness, and add to them when new issues arise. An SDET, or a rotating team member, or whatever. The average dev isn't going to write a test that's super valuable if they also wrote the code.

Like most others here in the comments, I find the results not particularly controversial. Pretty easy to imagine scenarios where lines of code are run, but the tests themselves are suboptimal or incomplete.

As an aside— I've often mused at to whether theres a more useful multivariate (but still coarse) measure that could combine coverage with cyclomatic complexity and number of tests/assertions. Seems like it would be incrementally better than just coverage alone.

> Well you executed the branch/line at least once with one potential input. Was it an edge case input or a happy path input?

How does that matter? If something about the input causes a difference in the execution of the code, then 100% coverage means you necessarily tests both kinds of input. You can't reach the edge case branch with the happy path input.

Now, if your code is just pumping data from one point to another point, it's possible that the destination will vary its behavior based on the input in a way that your coverage-based testing can't see. But if you're doing something with the input yourself, then 100% coverage means you tested every possible kind of input.

(Actually, a "branching" problem can still arise if you have something like

  if( condition1 ) {
    do_stuff();
  }
  
  if( condition2 ) {
    do_other_stuff();
  }
  
  if( condition3 ) {
    do_other_other_stuff();
  }

Because with this code, it's possible to get test coverage of every line of code (there are 6 of them, not counting the '}'s) while not covering every branch (there are 8 of them, depending on the combined truth values of condition1, condition2, and condition3).)

That's because you have such simple conditions. How do you compute them?

   if (x < y) { ... }

really changes things. Did you test for x === y? for NaN? for very close doubles that should have passed/failed for business reasons? For types other than numbers if your language allows?

Expressions can have any number of edge cases that code coverage can't account for.

Depends on the language/type safety offered. If your language has a NaN (javascript) and you can assert the expected behavior of x < NaN (false) or if you don't even want to encounter that, then you're able to reason back to the generation of the inputs. Wherever x and y came from, if there was a possibility of an NaN being a value, you want the generating function to exception or cast via some sort of validation before handing them off to other unsuspecting functions.

Well, imagine two different analyses of the code:

1. Does each branch do the right thing?

2. Does the code go down the correct branch in the first place?

I've focused more on the first question, and you're focusing more on the second. But it's fair to say that each question is a reasonable focus of testing, which weakens my comment above.

This isn't true at all. Consider buffer overflows, bugs in your bitwise arithmetic, division by zero, or null pointer exceptions.

In my experience, when a function has more than 1 if statement, it's probably hard to test and you might want to split it up.

When a function has has more than 2 if statements, you definitely want to break it up.

Imagine the function where conditions aren't related at all, as you posited:

    myfn(int x, int y, string foo) {
        if(foo == "bar") {
            do_stuff();
        }
  
        if( x < y ) {
            do_other_stuff();
        }
  
        if(time() == "noon") {
            do_other_other_stuff();
        }
    }

What's really happening here? Why is all this wrapped in 1 function, when the args aren't related at all, nor the work they are dependent on?

Let's say the caller was doing: myfn(1,2,foo);

I would split this function into 3 different function calls in the caller...

    possiblyDoStuff(foo);
    possiblyDoOtherStuff(1,2);
    possiblyDoOtherOtherStuff(time());

Let's use something that you wouldn't simply refactor upstream:

    myfn(string go, string for, string foo) {
        if(go == foo) {
            do_stuff();
        }
  
        if(for == go) {
            do_other_stuff();
        }
  
        if(foo == for) {
            do_other_other_stuff();
        }

        someOtherFn(go, for, foo);
    }

Move that complexity into smaller chunks:

    myfn(string go, string for, string foo) {
        possiblyDoStuff(go, foo);
        possiblyDoOtherStuff(go, for);
        possiblyDoOtherOtherStuff(for, foo);
        someOtherFn(go, for, foo);
    }

Now you have the 7 tests. 2 for each possibly = 6 which are pretty easy and 1 for myfn. It is exceedingly rare that functions require this kind of attention. There are usually a bunch of side effects or return values that are dependent on these checks. eg:

    myfn(string go, string for, string foo) {
        // examples of why you are calling these, to get vals
        var go2 = possiblyDoStuff(go, foo);
        var for2 = possiblyDoOtherStuff(go, for);
        var foo2 = possiblyDoOtherOtherStuff(for, foo);
        return { go2, for2, foo2 };

        //move this to caller someOtherFn(myfn(go, for, foo));
    }

100% or you suck. You need 100% and then some for FULL coverage.

I work with people who turn in 80% PRs...these people are bug factories.

If (as the study says) # of tests is the true correlate then

1. We need some way to compare # of tests across differently-sized codebases, i.e a percentage metric like coverage 2. The fact that correlation disappeared when controlling for absolute # of tests seems to suggest it's still encouraging people to write more tests. Once your org gets to a certain size you need these blunt instruments like minimum coverage requirements because you can't just rely on individual excellence anymore.

Ever since I developed code coverage tools at Apple in 1989, and tested them for Borland in the early 90’s, I knew and have been telling people in MY conference slides that code coverage is a nearly useless metric. Anyone who thought critically about it for ten minutes knows it’s nonsense.

The one thing code coverage tells you that is of any significant value is what you haven’t tested. You still know very little about what you have tested.

Meanwhile, this article continues the dopey practice of using the word “coverage” only with respect to code and not with respect to many other forms of coverage, such as data. Data coverage is not easy to measure, yet it is far more important.

Generally speaking, academics know very little about testing, and academic papers about testing are nearly useless. If you want to learn about testing from an academic, read sociology, philosophy, and cognitive science.

Yet it's extremely popular to have huge, inefficient test suites running on continuous integration servers hundreds of times a day. It boggles the mind.

This is a good thing if you consider the price of not having them. I have worked in shops without tests. It's a great way to hand out free money to unsuspecting customers.

Yes I agree - I have been a testing fanatic for the better part of the last 10 years, after being absolutely paralyzed at a company without tests. But, after all this time, I believe their cost-to benefit-ratio is horrendous.

It’s fairly common to hear of test suites with a 2:1 ratio of test to implementation lines. That would be fine if they didn’t immensely prevent refactoring and block merges / deployments.

Contrast that with something like F*, where the specification and code are right alongside each other, and the implementation gets proven. They are reporting a 5:1 ratio of specification to implementation code lines, which is much better than only a few years ago. Soon I think we’ll be striking distance to largely get rid of huge test suites which are so commonplace today.

The one thing is a spec doesn't mean the product is correct - you still need testing, just in a different way. It'll probably replace unit tests though.

A spec can catch anything your tests can. And if your spec overlooks something, your tests will overlook it too.

I've seen projects using formal methods including mathematical specifications miss things caught in end-to-end tests written by domain experts rather than formal method engineers. Perhaps you're right in theory but I've yet to see it in practice.

This just means that you thought of something in the test suite that wasn't in the spec. You can underspecify something.

Correct, there’s still nothing stopping us from writing the wrong spec or writing an insufficient spec, but there’s really no solution to that problem anyway. And the same problem exists with tests today - you can have an insufficient amount of test cases, or you can write the wrong cases because you misunderstand the requirements.

The difference being, when you realize your tests are wrong, what do you do? Change and add more test cases. You may also need to remove some, because they don’t make sense anymore.

Vs., updating logic in a specification, and that’s it.

I’m not sure what your point is about ratios. I don’t think they mean anything, strictly speaking. Perhaps they are related to some special knowledge you have that is not conveyed by the numbers.

Testing is a social act. Testing itself cannot be obsolete until complexity and misunderstanding is obsolete. This will happen never.

Perhaps certain kinds of shallow automated checking will become obsolete over time.

Possibly the most valuable post I've read on these news pages.

Damn, I love geeking out over testing.

> academic papers about testing are nearly useless.

You must not have read the post--it's principally summary of a paper by two academics at the Univ. of Waterloo.

I did read the post, and I am referring to the paper by the academics. What did you think I was talking about? Hello is this mike on?

> Anyone who thought critically about it for ten minutes knows it’s nonsense

Can you explain a bit more? I have only done coverage on my home projects. I get 100% every time because otherwise why bother. I started experimenting with 100% branch coverage. I understand sometimes there's a few lines you can't test like if there's a fork() but in my cases I was lucky enough to not need it and didn't need to exclude anything (I was thinking about excluding an impossible to execute default case since I did val&7 and tested all 8 cases but I ended up using if statements instead for that one off case)

It seems to me as long as your testing your own code its great. It gets annoying when testing an API that you can't ask to fail. I never mock anything. It seems like coverage is a great solution

In one project I used fuzzing data and another I specified valid date and say anything invalid is not a bug and may report incorrect results (I think I detected most/all bad data and rejected it which people hated)

> Can you explain a bit more?

Not the parent commenter, but i'll attempt to chime in: in addition to the default "dopey" practice of measuring coverage in terms of statements of code, in addition to data coverage as mentioned by the parent, you could also consider coverage in terms of possible execution paths through the code or coverage in terms of requirements. For interesting computational code (i dunno, consider a mixed integer program solver) it might be possible to get 100% statement or 100% branch coverage at the same time as getting approximately 0% data coverage, 0% execution path coverage, and 0% coverage of functional requirements.

Another perspective is how much value the test gives relative to the effort and cost to set it up and maintain it. You can have 100% unit test coverage (statement, branch, data, whatever) but maybe your application doesn't even do anything when it boots, because you forgot to call any of the units. So if effort and time is limited, you might get more return on investment from writing some good integration tests or hooking up a fuzzer than focusing on hitting the somewhat arbitrary but easy to measure 100% statement coverage number.

Also, like everything in life, focusing on a single objective like "statement coverage" and trying to maximise it can lead to strange outcomes. There was a fascinating blog post / war story somewhere describing a software system architecture where the organisation mandated non-negotiable minimum 80% statement coverage for unit test suites. This system had some pretty low level code and a bunch of that code wasn't possible to test in unit tests. So how do you hit the target? Well, if you add a few useless layers to your application architecture that are easy to unit test, then you test them and leave the hard code that interfaces with the real world / hardware untested, and you hit your mandated target, but the overall level of software quality is worse.

well said

Do you know what states are? Of course you do. So then you know that executing a line of code in one state could cause behavior that won’t occur in another state. Code coverage doesn’t say anything about state coverage.

You’ve heard of table lookups? Of course you have. So then you know that different results will come from a lookup that is based on a different query or hash key. The lookup is one line of code. But maybe you visit the code hundreds of times before you hit upon the result that gives you bad data which tanks the rest of your algorithm.

Come on, man. You know what I mean. Another example: I once wrote a table comparison routine in ObjectPal. I tested it and it seemed to work, but the first time I compared a table with embedded bitmaps I got “Object of type bitmap cannot be compared with <> operator”

See what I mean?

Testing is an art that takes quite a while to master. People are able to write tests but many don't know "how" to write tests. Testing is extremely valuable but at the same time very easy to get burned. When testing done wrong (like abusing tests for coverage):

1. It makes the code too inflexible to refactor, leaving the system too rigid to grow.

2. It makes people frustrated on how to write tests in this project.

3. It makes people spend more time on writing tests than implementation.

So some quick tips:

1. Only test the behavior from the user/consumer's perspective, do not test implementations. (Therefore the internal can be refactored without deleting tests.)

2. Be simple, clear, and consistent about how to write test on the project. If it's layered architecture, be clear about which layers need to be tested. And be minimal, two layers of testing is usually more than enough. Make sure the testing rules is enforced in the code base in the beginning, and then it will be downhills since people are mostly copying.

3. People always talk about the Testing Pyramid: the idea is coarse-grained tests are valuable but too costly while the fine-grained tests are cheap, so it's all trade-off and the optimal is to make them like a pyramid. But in reality, certain layers of tests could be drastically optimized. The idea is maximize the granularity on behaviors covered, while minimize the infrastructure involved (e.g. carefully designed monolith + hexagonal architecture), or make the infrastructure fast and parallelizable (e.g. SQL sandboxing). The end result is a certain layer could be a better trade-off than others, so concentrate tests there instead of writing tests repeatedly.

> 1. Only test the behavior from the user/consumer's perspective, do not test implementations. (Therefore the internal can be refactored without deleting tests.)

Sometimes you still need to test implementations. We had database writers/readers that automatically retried on failure/disconnect/failover/whatever and you can't test those by testing the API.

Thanks for pointing it out. I was not clear that I was referring to the tests related to business requirements (which is the majority of the tests).

Infrastructures could be treated like third-party modules as they could be tested like any open source project separately.

Aiming for 100% test coverage actually produces negative value.

You don't need a "study paper" to know this. Just work with a team that aims for 100% coverage for a few months and you will see it for yourself.

The negative value comes from:

1. The time wasted on writing all these tests that are mostly ceremonious in nature.

But, more importantly:

2. It makes refactoring a big pain in the ass.

Why? Because 100% test coverage forces you to test the implementation details of everything.

So, every time you do a non-trivial refactoring, you will get tens or hundreds of failing tests. This is mostly noise. If there's any signal in any of these failed tests (signal = information that you screwed something up during the refactoring) it will be very difficult to catch.

What do you do when you refactor something and break a 100 tests?

You delete all the failing tests and start again: look at the coverage report and start writing tests for the portions of the code that are not covered.

It is very unpopular view.

I have never been a fan of unit testing and instead I prefer end to end functional tests -- where you write tests that verify your application still behaves exactly as expected but does not care how it is implemented.

This usually requires much less code, does not deter refactoring and also focuses on the one thing that is really important for the client.

Since the outside interface of the application is less likely to change compared to the implementation, the tests tend to be more stable and require less maintenance over life of the application.

Unit tests were supposed to help refactoring (by making it easier to ensure the code still works after change). The sad reality is that I can never trust unit tests. I still need to research the code around the function I am modifying to be sure I am not breaking anything.

Also my personal style is to do shit ton of refactoring. This means I usually start with something that only vaguely resembles the end result. I spend a lot of time moving stuff around until I get rid of everything that I don't like.

This absolutely precludes writing unit tests up front. But also, after I have spend so much time polishing the code, writing unit tests for it is absolutely the last thing I want to do. There is absolutely no possibility that people will put effort into writing tests well if they did good job writing the actual implementation.

Yet another reason why unit tests are broken is that there is no mechanism, no feedback loop to ensure quality of tests.

When you write application code, if it is broken you will have feedback in the form of defects and outages or various other problems.

One reason that documentation is usually broken is because there typically does not exist mechanism that would ensure that documents stay in sync with design and implementation.

The same problem with unit tests. The only strong signal you get is when the test does not work. But there is no good signal to fix real important problems like missing tests for important part of the contract.

> Also my personal style is to do shit ton of refactoring. This means I usually start with something that only vaguely resembles the end result. I spend a lot of time moving stuff around until I get rid of everything that I don't like.

Same here.

I do think there's value in some unit testing: mostly for small units whose behavior is a bit tricky to verify by just looking at the code or running the application. These are usually functions with tricky mathematical expressions, or something like that.

So true. Testing and documentation "freeze" a solution. Going too deep with them from the beginning has a similar effect to early optimization. There is an economic factor in which code is mature enough to invest in testing/optimization/documentation, before that point effort is wasted.

Agreed. In general, I prefer integration tests, especially end to end tests.

The one exception is for utility functions. I advocate TDD for utility functions, with heavy testing of standard and edge cases, to ensure that the functions conform to spec and any future tweaks / refactors of the functions continue to conform to spec.

3. Your code base may start to become contorted. I've seen good programmers create bogus classes to allow test-time mocking, or add oddball env vars and configurations to let the test harness manually reach every last line. Even if that line is not worth testing:

    if(!(x=malloc(BUF_SIZ)) || ENV[TEST_MEM_FAIL_12]) {
        exit(1);
    }

Tying code and tests this tightly discourages refactoring.

Another example: a different code base that I helped on had some insanely long one-line conditionals that anyone would break into multiple lines ... but doing that would have blown the "100%" coverage.

"Mocking" is such a weird thing to me and I don't think it serves a good purpose. It's the kind of thing that would only arise if you assume a-priori that 100% test coverage is a non-negotiable must.

If you have a function A that calls B to get some data (by doing I/O) then process it using C, then the 100% cov rule would force you to mock B when you test A. But then what is the value of this test? What guarantees is it giving you? You are basically testing the internals of A, you are testing the implementation details. You are testing that calling A is the same as calling B and passing the result to C.

Mocking and coverage aren't that closely related, but I agree that mocking makes tests less realistic, and therefore less useful.

Mocking can be helpful when that downside is still better than the alternative, e.g. if 'real' calls have significant latency, or a significant monetary cost, etc. In those cases mocking lets us do a whole lot more testing for the same amount of time/money/etc., which can make up for the loss of realism.

Mocking can be especially useful when adding tests to a legacy system. Greenfield work should aim to minimise the amount of mocking required, ideally to zero.

Mocking helps with architecture, though.

When A calls C through B, you have a tight coupling between A and C, but mightn't be aware of that, because you only see A calling B.

Lets say C reads a CSV file. By testing implementation only, you might 'create a csv file' then call A.import() and assert some records in, say D are created. By testing A in isolation, you make apparent that it couples to C and D because you have to mock them out. B is a direct dependency of A, so you don't mock that. At the least, you have now documented the accidental coupling. But at the best, your tests showed you a design issue.

You don’t even need to do a study to arrive at this conclusion. Has no one read Djikstra’s writings? https://www.cs.utexas.edu/users/EWD/ewd02xx/EWD249.PDF

“Program testing can be used to show the presence of bugs, but never to show their absence!”

Everybody in our industry wants to bash math and say that anyone can write programs, but programs are logical systems and can only be fully understood with math and logic.

We have the tools for understanding and reasoning about infinitely large structures, programmers just refuse to use them, and even deride them.

Well, I am not in that camp, and I work on math and logic skills to help me develop working software. It isn’t perfect, but it’s the only path that makes sense to me.

> But complexity sells better and the market pulls in the opposite direction. I still remember finding a book on how to use "Wordperfect 5.0" of more than 850 pages, in fact a dozen pages more than my 1951 edition of Georg Joos, "Theoretical Physics"! It is time to unmask the computing community as a Secret Society for the Creation and Preservation of Artificial Complexity. And then we have the software engineers, who only mention formal methods in order to throw suspicion on them. In short, we should not expect too much support from the computing community at large. And from the mathematical community I have learned not to expect too much support either, as informality is the hallmark of the Mathematical Guild, whose members —like poor programmers— derive their intellectual excitement from not quite knowing what they are doing and prefer to be thrilled by the marvel of the human mind (in particular their own ones). For them, the Dream of Leibniz is a Nightmare. In summary, we are on our own. But that does not matter. In the next fifty years, Mathematics will emerge as The Art and Science of Effective Formal Reasoning, and we shall derive our intellectual excitement from learning How to Let the Symbols Do the Work.

Calculemus!

https://www.cs.utexas.edu/users/EWD/transcriptions/EWD12xx/E...

> We have the tools for understanding and reasoning about infinitely large structures, programmers just refuse to use them, and even deride them.

Could you point out which tools you're talking about?

Well, by this I mean math (and mathematical logic). Math is the tool for reasoning about possibly infinite concepts, i.e. you can make a statement about infinite sets and still know if it’s true or not. We aren’t limited to what we can see and touch, which is good because any non-trivial software application is so large that it could never be drawn out like a building blueprint. It can only be described and reasoned about abstractly.

But, you are probably asking about what ‘tools’ can be applied to programming, meaning some kind of library or application. These are also out there. Here’s a few that I think are promising:

TLA+: This is a specification language for describing and reasoning about computations as state machines, with a particular focus on modeling distributed systems. It has been used at Amazon to check designs for their distributed algorithms. They have used it to check parts of S3 for example: https://lamport.azurewebsites.net/tla/formal-methods-amazon....

Best reference is the book Specifying Systems: https://lamport.azurewebsites.net/tla/book-02-08-08.pdf.

Then you have theorem provers / proof assistants based on type theory. These are frankly complex, but getting better.

F*: This one is the most exciting to me. It’s currently being used to develop a formally verified HTTPS stack: https://project-everest.github.io/. They already have components released to the Linux kernel and Firefox. One of the most exciting things about this tool is that you can verify an efficient, stateful algorithm and extract highly performant C code from it. Their verified implementations have equal or better performance to the current solutions out there. So all of the overhead is for verification, none exists at runtime.

Of course you can’t talk about proof assistants without mentioning Coq. Its claim to fame is producing a formally verified C compiler, CompCert: https://compcert.org/.

As you can see, so far, formal verification has been mostly limited to components of systems, not entire systems. But, it’s a start, and the scope of what we can verify is getting larger.

I know about all the tools you've mentioned as I've been following the space from a distance, was just curious if you had a more "outside research" approach in mind when you claimed developers refuse to use the available "tools"... I haven't seen Coq, or even Idris or Ada+Spark, which arguably much closer-to-the-industry tools, being used for anything but very niche cases (like the case you mentioned about using TLA+ at Amazon).

I am probably one of the developers who would gladly use anything like that, but from my brief experiments and knowledge, they are very, very far from becoming usable in a general setting, so no, I don't refuse to use the tools, the tools are just not good enough to justify the high cost of using them yet.

It boils down to two camps: open to formal methods, and not open to formal methods. And, I would say the overwhelming majority of developers, especially here on HN, are in the "not open to formal methods" camp. I can find comments that justify that, there are also data points from speaking with coworkers and colleagues. So that is what I was talking about when saying that logs of people "refuse the tools." They just aren't even open to them to begin with.

You sound open to them, but dissatisfied with the tools themselves. I'm honestly in that camp too! We are not there yet in terms of being able to use this as the primary way that an entire application is developed. But, I believe that's the direction we should go in. You're right - we really need a tool that wins people over with the cost to value ratio. The promise of something far off in the future isn't enough for most people.

This makes perfect sense: a simple function with two code paths that splits on the comparison of two signed integers immediately requires a minimum of three test cases for correctness, yet it only takes two to achieve 100% code coverage.

Checking for correctness for corner case values - maxint, minint, zero - adds a minimum of another 9 cases.

And it will take many, many more test cases if you're working with a weakly typed language and you're potentially comparing an integer with a floating point value. Or strings. And so forth.

Not to disagree with the idea that you generally need more test cases than control flow paths to really test correctness well. Just a question about your example -- let's say your requirement is a function that does this:

  void fn(int a, int b)
  {
    if (a == b)
      printf("equal");
    else
      printf("not);
  }

What are the 3 test cases you would write? What are the 9?

  fn(1, 1) -> "equal"
  fn(1, 0) -> "not"

What more useful tests are there here? Other than just picking arbitrary numbers, maxint/minint/1/0/-1/etc are not significant edge cases here.

It’s not a problem here, but if the type doesn’t guarantee reference equality for equal value, you’d also need a test to differentiate between

    a == b

And (in Java)

    a.equals(b)

In order to catch misguided refactorings. (I’ve introduced bugs this way before I learned that == can give a false positive for strings if the strings are interned)

in java the method that takes two `int`s would still use ==. java has value types which don't need `equals`.

Yeah, that’s why I said “it’s not a problem here”.

Let's say I mistype 'a == b' as 'a <= b' when writing fn (the purpose of tests being to validate that the code is correct, after all). The test values of fn(1, 1), and fn(1, 0) will still display "equal" and "not", even when the function is obviously incorrect.

Let's add 1 to a before the equality comparison in order to meet a new business requirement. fn(1, 1) still works, but does fn(maxint, maxint)? Or does it suddenly throw an exception (or worse, silently roll over to minint)?

Right but you could also accidentally mistype it as 'b == 1'. Or 'a == b*b'. Or '(a == 0 || a == 1 || a == maxint) && a == b'.

I'm not saying more tests can't catch specific bugs you might come up with, I'm asking how you can choose numbers that have fundamental edge cases for this specific requirement without looking at the actual implementation. I don't think you really can. maxint/minint/0/-1/1 are generally common choices but only because they do tend to catch signdness errors or inequalities or division by zero or whatever, they aren't really fundamental to the requirement of this function though. If we just wanted to blindly blast those numbers in, it would be 25 individual test cases for all permutations. Doesn't seem to be reasonable.

Is it reasonable? It depends on your goals. Do you want to ensure that, for the set of well known edge cases, your function is correct?

Proving the function is correct for the known edge cases for integers and integer comparison is not excessively hard. Let's go for the worst case of edge cases, and:

- Pick the input pairs that are problems with ints: the set of (minint, maxint).

- Pick the input pairs that are problems with comparisons: the set of (-1, 0, 1).

- Pick input pairs that show that it's correct where a < b, a = b, and a > b. That last set of cases is nicely covered by the previous two.

That's a total of 25 input pairs(5^2). In my opinion, that's not an unreasonable number of test cases to test for correctness. Why do I think it's reasonable? Because while it's technically 25 test cases, it's really just one test that runs through an array of input values in a loop. Really simple, and adding more test cases as the business logic changes is equally simple.

And let's be honest, given how quickly computers work these days, even testing all ints for both inputs isn't completely unreasonable (though I wouldn't make it a part of a unit test suite). Especially if your logic is more complicated than a simple equality test (which most function logic will be).

EDIT: "Accidentally mistype" 'a==b' as '(a == 0 || a == 1 || a == maxint) && a == b'

Love it.

I'm not sure that it is reasonable. It doesn't scale up to anything but the most trivial functions. It also does not prove the function is "actually correct", that is a fallacy. If you want that, you have to use formal.

The problem is not this one particular function, it's transferring this methodology to something more complex. A reasonable set of tests that does not aim to prove correctness is quite entitled to assume a reasonable implementation. You wouldn't have to arbitrarily re-run all your test cases with values incremented by 42 just in case someone incorrectly subtracted 42 somewhere, for example.

You're testing for the characteristics of your function - inputs and their outputs. If your function takes strings, there's a well known set of strings that are typically known to cause issues. It's the same for every data type. And for every operation against/between those data types, there's a known set of issues.

You can be pretty certain of a function's correctness (or more accurately, the correctness of a refactor - what unit tests are most useful at, in my opinion) without having to re-write the function in a slightly different way.

But to get back to the topic at hand - code coverage is a pretty bad measure of program correctness. What I was really trying to show is how bad code coverage is at predicting correctness for even the shortest and simplest of functions.

I know that's the idea, what I haven't seen is any real data showing that is a significant improvement in testing. You pick a few things, there are a lot more that can go wrong, off-by-one can come up in comparisons easily, shifting left or right can happen easily, assortment of bitwise operations can happen. And that's just integers. When you get into floats what are you going to plug in all common math constants as well as a range of values around each of those, various products and summations and factors of those things? No of course not. It's perfectly reasonable to simplify the test based on what the function is supposed to do.

I'm not saying don't test any integer corner cases, I'm asking for the reason why the original poster says including those is good (and listing only 9 of possible 25 combinations to boot). In any case, why is that set of tests the golden one? That was just asserted as fact without any real reasoning I could see.

If you tell me those 25 are the right set of tests, I can say you're wrong because 2 is also very common in computing so 2 and possibly -2 should be tested as well. My 49 tests are clearly superior. Then someone comes along and says well 10 is common in scientific or business logic, so better add 10 in there as well. Using the same arguments as you did to reach 25 we can now get to 81. And so on. Apply it to a function with 5 arguments and we're up to 60 thousand tests. Not reasonable.

Consider the input as a big space of points, where each point is a possible input. These points aren't unrelated; speaking loosely, foo(3.000000000001) is usually more likely to behave like foo(3) than foo(4) is.

The different behaviors on this space of points are due to the control flow and mathematical expressions within the function. You could thus model the function's behavior as a partition of the input: the different sets of inputs that will behave "identically" WRT the control flow inside the function. You could probably formalize this with a suitable notion of continuity, but I haven't bothered.

Accordingly, testing corresponds to searching this space for problematic partitions. It's a bit like playing battleship: if you know a lot about the possible shapes of errors in this mathematical space, then you can choose a few small points that can totally prove their absence.

Even if you don't know the shapes though, by choosing your points carefully, you can rule out shapes that are "sufficiently large". This is the impetus behind boundary testing; you don't prove that errors don't exist, but you can prove that they are relatively "pointlike"; that there aren't arbitrarily wide swaths of the input space that all exhibit the same erroneous behavior. Since the input space is usually vastly multidimensional, there's usually pretty hard limits on how much you can actually reduce the set of possible "problem shapes," though.

A lot of in-practice software testing out there (I can only really speak about the work I've seen at Google) is statistical in nature: you run a huge amount of tee-d or saved production traffic through a service, and run a massive diff of the output, after painstakingly making your service deterministic. By doing this, you can show that a "typical" query is unlikely to cause a "catastrophic" failure.

Now you understand - testing is extremely close to useless. You can't perfectly choose the test cases for code without looking at its implementation. And by that point, you are just testing the implementation and working backwards.

What you want is a specification of the behavior, and to verify that the implementation satisfies the specification. This will be proven in the general case, and then you don't need to think about individual cases.

I don't think testing is close to useless, I think it has a lot of value but trying to prove general correctness with your test cases is a fool's errand. Adding test cases intelligently for tricky cases in the specification and/or implementation, hard-to-hit error conditions, or adding tests for bugs you previously fixed, etc is perfectly reasonable without doggedly chasing 100% coverage or 100% correctness.

And yes I'm aware of formal methods, and agree they seem like the better way to go for proving.

I wouldn't even write this way because the else confuses what's happening.

    void fn(int a, int b) 
    {
        string ret = "not";

        if (a == b)
            ret = "equal";

        printf(ret);
    }

*I understand, this is not a good optimization in every language.

Your code is far less clear to me than the above code.

Though I'd probably write:

  void fn(int a, int b) 
  {
      printf(a == b ? "equal" : "not");
  }

How does the 'else' confuse what is happening?

The function will perform a printf, so nesting it in an if-else subconsciously implies it's optional.

This is a quirk of human text comprehension. In a small program, it seems arbitrary because you can see it all at once. Make the branches do more than simply printf and you can see why you want to frontload the else, rather than pick through the if/else to see what always happens.

I can kind of see what you mean, but if you make branches do more then it's also quite possible the common actions become more tedious to factor into common code. And I don't think it's good practice to read control statements like that (obviously indented loops can't be assumed optional either).

  int pow2(int v, int pow)
  {
    int result;

    if (pow >= 0)
      result = v << pow;
    else
      result = v >> -pow;

    return result;
  }

Looking at this, both sides of the if/else perform a multiply by 2^pow, you can't take the operation to possibly not happen.

So I agree factoring common operations can be a good thing, but I think it's really a case by case basis and my code is fine.

I would also caution against assuming something may not happen elsewhere in a function just because you see it inside a conditional block. This happens frequently to various degrees and I've never encountered a style policy preventing it or any compiler or linter warnings for it.

IMHO the original version without the mutable local variable was much more obvious. In any language.

Though I would also accept:

  void fn(int const a, int const b)
  {
      printf(a == b ? "equal" : "not");
  }

to avoid duplicating the call to printf(). If this were Rust I might suggest:

  fn theFn(a: isize, b: isize)
  {
      let ret;

      if a == b {
          ret = "equal";
      } else {
          ret = "not";
      }

      println!("{}", ret);
  }

but only because Rust, unlike C or C++, will enforce that `ret` is assigned exactly once before it is used.

> IMHO the original version without the mutable local variable was much more obvious. In any language.

This is not about obviousness. You can write the same expression lots of different ways, which are all obvious. This is still a question about testable paths and code hygiene.

Nesting the same function call in both branches of if/else unintentionally obscures that it's not a conditional printf call.

If you're going to write:

printf(a == b ? "equal" : "not");

you don't need the function at all, completely missing the point.

Your second example is definitely better in Rust.

> Nesting the same function call in both branches of if/else unintentionally obscures that it's not a conditional printf call.

And the version with mutation masks the fact that the initial value assigned to `ret` may or may not be used, which seems like a very similar problem to me. Plus it involves mutation, which IMHO significantly increases the complexity of any code. Unfortunately C offers limited options here; as cryptic and controversial as it may be, the ternary operator is the only (non-hackish) way to make an expression which selects between two or more values based on a boolean condition without either the possibility of failing to assign a value or requiring a "default" value which at least some paths through the function won't use. (What if there is no reasonable default value and all the required branches are expensive to compute?)

> If you're going to write: … printf(a == b ? "equal" : "not"); … you don't need the function at all, completely missing the point.

The function still encapsulates the specific comparison and the way the result is reported—exactly like the other versions—so I wouldn't say it's completely useless despite containing only one line of code. This is, of course, merely a toy example. You could assign the string to a named const variable if you wanted to make it slightly more self-documenting, though I question whether that would make this particular code any more readable.

This affirms my intuition that the power of test suites arises from their coverage of the data-cases, and call-sequence, and not simply from "visiting" more lines of code. This also is likely the underlying reason for the extreme effectiveness of fuzz-testing and property-based testing.

They go hand in hand. It’s obvious that missing branch coverage means your data cases are not exercising all of the edge cases in the code.

Not all code is reachable. Especially things like top-level try/except clauses.

They're in your source code - they are reachable.

From the entry point, stub a function to throw an exception. You'll reach it. Unit testing isn't about finding every path. Nor is it about writing unit tests for every possible combination of values in a program. A java function that uses an int does not need to test -2147483648 to 2147483647 as inputs. That's not helpful.

That’s an interesting example, that’s exactly where you’re likely to find bugs - at boundary values. You should test those more than you should test the number 5.

This is exactly why unit testing gives a false sense of security. You’ve done a lot, you’ve written all kinds of tests - but at the end of the day, you don’t get a proportionate amount of confidence about the code because there are infinite more cases that you haven’t thought about.

In Java, if my function doesn't modify the int, there's no reason to test the boundaries, other than 0 if the int is used in a calculation. The type system doesn't solely determine what tests to write. Tests are inferred from a combination of type system, timings, statements and variable usage.

I've been asked many times what is the right code coverage percentage to aim for. I've also asked this many times in interviews, to tease out a discussion. My answer is: you need full coverage. I then continue to explain that I didn't say 100%. Although I believe that the average Java microservice (which is the general sphere that I move around) can easily achieve >98% coverage. Easily. But "full coverage" means that all reasonable scenarios should be covered. No code coverage tool is going to measure that for you (I think). Reasonable scenarios are... everything? Time is finite, so most teams will never be exhaustive, but one can try. The "80% coverage crowd" is throwing in the towel before even starting.

I've occasionally been lulled into a false sense of security by the difference between "every line is run" and "the function is fully tested".

On the other hand, even if I'm half-assing the unit tests, and implementing the bare minimum necessary to reach whatever arbitrary coverage% is being sought ... I still find bugs, bugs that almost certainly would have a real impact in production, even if I already ran an end-to-end test and was pretty sure it was working fine[*]. Not every time I write tests, but often enough to feel like it pays for all of the time "wasted".

So even if coverage metrics maybe arbitrary, I don't really begrudge having to meet them. And honestly, once you're forced to start writing the tests just for coverage, it isn't that much extra work to make them a little more comprehensive than the floor set by coverage.

[*] Honestly I'm shocked at how well buggy software works, for the most part.

That's my experience, too. Coverage isn't a good proxy for test quality, for sure, but it's a good proxy for having looked carefully at every line. I think coverage testing is useful as long as you're prepared to throw those tests away when you refactor, or at least ignore them during refactoring.

Agreed - if you can test it then test it. 100% coverage may not be easily achievable with some languages/frameworks, but as other have pointed out, it's a good idea to work towards it for dynamic languages so you can reduce the possibility of runtime errors.

I usually just use Unit Test coverage as extra insurance that I didn't accidentally forget a path when writing up tests for new classes, or when trying to assess others tests to see if/what they miss. Outside of that, I basically ignore it.

That (per)mutation testing, sounds like pitest, which I've had fun with using to gauge the effectiveness of tests I've written in the past.

A team getting to 100% test coverage and enforcing it feels like an application of Goodhart's Law.

When a measure becomes a target, it ceases to be a good measure.

At my last job, we required 100% code coverage for most code, and the other parts of the code (in ideality) were marked with ignores that were well-thought-through.

In practice, I ended up writing a bunch of test cases only to hit the if blocks. It didn't mean that the tests I wrote were any good. I'm still a fan of 100% test coverage, but with some leeway for what needs to be ignored, and what needs to be tested later (non-critical paths, if you have a deadline, perhaps)

It gets crazy with demands for 100% test coverage, in languages where you're encouraged to write compile-time logic, macros, setup-dependent minor details, time bombs, asserts which throw compiler errors when you misconfigure, etc.

I have never used the % stat as anything more than curiosity but I have certainly looked at the highlighted reports generated to see large blocks of untested code which is genuinely useful info to see.

Beyond a certain point, test coverage has diminishing returns.

You end up testing failure cases which are better addressed by good error handling. For example, inability to open a file due to file permissions can be covered in a more general way.

Erlang is one of the most effective platforms for making reliable software. It has the concept of "supervisors" which will monitor and restart code if there is a problem. This handles both known problems which are appropriately managed by the standard strategy as well as unexpected problems.

You can also use other static code analysis tools that ensure that you have dealt with every possible return code from a function.

Other things become more useful than testing, e.g. implementing observability to tell you what is happening at runtime.

Instead of more unit tests, I would prefer to have more validation checks when deploying code and in production. A validation check for Blue/Green deployment prevents bad code from going live. A production check ensures that nothing bad is happening that affects users, e.g. if we would normally get 10 signups per hour, and we are now getting zero, then it's time to page someone. I will take that over an extra 10% code coverage any day.

Put the effort that you were going to use on code coverage into something that gets higher value.

In practise, doesn't increasing the coverage highly correlate with increasing the test suite size, therefore proving the effectiveness?

Conversely, I struggle to think how coverage could be increased significantly without increasing the test suite size in reality.

It depends. When you tell developers "you must have 100% code coverage" they usually write tests that don't actually validate any functionality and instead get into every if block. Tests are useful when they test edge cases and assert behavior.

I've told this story many times before but at a previous job a senior engineer told me "100% code coverage is useless and you shouldn't go for it" but since he was being dogmatic and not actually thinking about what he was saying he was arguing against something very sensible. I was testing an expert system where everything was large if/else trees encoded in types + configuration. I wanted to make sure I tested all edge cases and activated all of the blocks when they made sense.

I had to fight for that extra coverage and it was, in the end, a massive help.

100% code coverage is still not a good use of everyone's time in most projects and languages. There's a bunch of trivial code that even feels wrong to test. Spend more time cooking up edge case tests that execute some of the branches way more than once.

Think of your code like a heat map. Higher heat on lines that get exercised more often by your tests. It's fine that _some_ code has no color at all, while you some other paths to be bright red in the end, instead of always just going for a uniform orange for everyhing.

> 100% code coverage is still not a good use of everyone's time in most projects and languages

The key here is most. Think about your use case before applying anything you read online.

> It's fine that _some_ code has no color at all, while you some other paths to be bright red in the end, instead of always just going for a uniform orange for everything.

This was the logic tree of a device used for health care work. Cost of failure was high. I had very good coverage of all edge cases that other systems could produce and tested a lot of extremes + a DSL for describing the input state.

The important thing here: an expert system is not most programs and the cost of a failure should really drive what your testing methodology is.

I did write most on purpose, instead of all.

Looks like your case is one where 100% coverage is a good thing and the cost paid for it is absolutely acceptable - nay needed to be paid. And you didn't just go for 100% and stop there because you hit some metric but you actually did the thing that's more important too ("data coverage"). Kudos!

If it takes you 3 days to write a test that covers a once-in-a-million condition and your service gets 2 requests per day, it will take you about 500,000 days to hit that condition once.

You've increased test coverage, but was it effective? Eh probably could do something more useful with your time.

(yes this is a contrived example, adjust numbers for your situation)

Depends on what the consequence of that failure might be. It could be anything from completely unnoticed to ending somebody’s life, rarity is only one dimension of risk.

    test("when a metric becomes a target, it stops being a good metric", () => {
      runApp(); // look ma, lots of "coverage"!
      assert(true, 'No errors!');
    }); // unfortunately paraphrased from real code

This test will inflate the test coverage, but it is a valid smoke test (assuming that any unhandled exception will cause the test to fail).

You shouldn't write tests like this; there's a high likelihood that the test will be flaky or not representative enough of production, and if the test fails, you often get completely non-actionable error messages.

If you just want to know that your app is broken, it's far better to monitor your live app (or staging environment or deployment pipeline or whatever) since that monitoring infrastructure can then be leveraged to collect other runtime health data in a more granular fashion.

Coverage only indicates which parts of the codebase were touched by the test suite. A big test suite size doesn't mean high coverage.

Coverage can be increased without increasing the test suite by reducing the code base size (within pratical limits obviously)

Personally, I only find coverage as a good indicator of which code still needs to be tested, like forgetting some edge cases or conditional branches.

Quality of trests is important, if you’re chasing 100% you’re liable to be writing inconsequential or incorrect tests just to achieve a number.

The correlation obviously exists for low levels of coverage.

Above 80% or 90% it becomes a poor measure.

I find it somewhat rare that I update code and replace an implementation in a way that tests immediately pass. That said, I think the value of unit tests isn't so much in the coverage metric as much as 1) showing someone did the legwork to test the code 2) document some amount of caveats and expected behavior, and 3) provide the next person to edit the implementation a small test "framework" (mocks, dummy data) to build on.

This.

For some codebases I think tests breaking at every change is the expected behavior.

In a way it gives an approximation of the dependency graph of the code worked on, and acts as a reminder of the other functionalities that one might not have intended to touch.

The tests provide a useful confirmation step. You make a change and the tests go "Hey did you know that this affected the output in these 3 places?" and most of the time you did know that and update the tests, but often you see a change you didn't actually expect and can now look deeper in to it.

I have also found immense value in tests that just check that literally nothing has changed. Like one that checked that the columns of a db table have not changed and when they have, the test reminds you that you need to decide if the new column is blacklisted from a certain feature and then add the column name to the test so it passes again.

As the suite approaches 100% coverage I'm not surprised coverage doesn't correlate with effectiveness. However on the other end, as the suite approaches 0%, I think the correlation is much stronger.

I view my job as knowing when to stop.

High coverage does not necessarily imply test suite effectiveness, nor does it even guarantee that your test suite provides positive value.

But 0% coverage does tell you a lot about your test suite effectiveness.

Also, beware the person too eager to proclaim that high test coverage is not necessarily good!

Last time someone told me this was discussing process maturity during a job interview. I let the team slide with this answer. Found out their coverage was effectively 0 and their process maturity was "break everything until right before each quarterly release" and I was out of there before my first week on the job.

That's a shame. You could have gained new experiences and taught a team something new.

Writing a program is easy and done a million times a day. Fixing a department is hard and people rarely get an opportunity to do it.

The Internet is a deluge of information and training material on this subject, so the teams that want to teach themselves this skill have already done so, leaving the teams that don’t want to do so.

I didn’t go into detail in my initial description, but this team misrepresented their process maturity to the extent that I had no qualms about bouncing out just as quickly as I got it to take a candid look around.

I’ve beaten my head against the wall for years on teams that didn’t give shit, so made testing and process maturity my primary focus during my job hunt, and made that clear to this team during the interview.

I was on a team that needed to write tests to deploy their app on an "enterprise CRM" that you probably all have heard of. You couldn't upload the app for distribution unless you had X% test coverage. Of course, nobody wants to write all these tests.

We had a few real tests. But most of our tests would call methods and check for != null. This worked great, got us up past 70% coverage with little effort. Supposedly these apps went through a "review" of some sort before they were released to the public. Oddly, our test methodology was never an issue.

  function (int i) {
      return 1 / i;
  }

100% coverage means nothing here when i = 0.

Furthermore, a line of code is arbitrary, and so is this metric.

Well, coverage can still tell you that this line is not covered at all. The absence of coverage is a signal, but the presence of it is not sufficient to prove that the code works for all inputs.

We’ve also known this since forever, this has always been Djikstra’s message to software developers.

100% coverage is baseline, not the ceiling.

kind of a bad function, though - can return -1, 0 and 1 only - plus the integer division by zero, unless 1 is a float type then 0 is positive infinite which is good.

Hillel Wayne discussed the paucity of decent research in software engineering land: https://buttondown.email/hillelwayne/archive/i-ing-hate-scie... (discussion: https://news.ycombinator.com/item?id=27892615)

My personal experience working on frontend is that it's often fastest to get everything looking right in the browser, and then go back and write tests. There is a ton of manual testing involved (i.e, getting UI into all its possible states) that just goes out the window when writing unit tests because you can't easily test for stuff like the correct whitespace between elements.

But then react-storybook and Happo came along (I'm sure the concepts have existed long before). I can actually commit the states I was trying to test to the repo (as stories), and CI can render them and take screenshot diffs with Happo; these are now the unit tests. So now the work I'm doing during development doubles as the tests as well, and there's no second pass required.

My first exposure to this was really eye opening. All my career I've been getting preached to that I should personally write more tests, and never stopped to think how I could make "developing" and "testing" the same thing. I'm imagining a magical tool someday that records all the things I do during manual testing and turns them into unit tests.

> But then react-storybook and Happo came along (I'm sure the concepts have existed long before). I can actually commit the states I was trying to test to the repo (as stories), and CI can render them and take screenshot diffs with Happo; these are now the unit tests. So now the work I'm doing during development doubles as the tests as well, and there's no second pass required.

Ohhh. Thanks, I didn't know this was possible (the committing of states, screenshots I've done before). I'm still in the hell of manual-testing the front end on multiple browsers and resolutions to check that changes don't break anything.

I've found that architectures like redux which force the separation of logic and side effects (eg file writes or networking) allow me to write robust tests against the logic code without mocks or stubs. My approach is to write enough code that I have a good idea of what the interface to SUT should look like. Then I write the failing test and make it pass then write another test... The tests I write generally only change when the interface changes or when the requirements change. It doesn't make the code too rigid b/c I'm only writing tests against the interface of this big blob of code. I'm free to refactor away and my test will still run as long as I don't change the interface.

Well, the number of case combinations that can occur can be exponential in the code size, whereas coverage is linear.

Say we have a sequence of five one-branch if statements: if (cond_i) { do_stuff_i } for i from 0 to 4. We can get complete coverage by executing that sequence just once with all conditions being true. But there are 32 distinct pathways through it, which could have all sorts of bugs.

Coverage also doesn't test for correctness at all; suppose that the one case that hits 100% coverage by executing every do_stuff_i produces the wrong result for that case. 100% coverage was still achieved.

A test suite consisting of nothing but a coverage test is ineffective in doing anything other than showing that the software doesn't bomb or infinite loop.

I think it's easy to conclude code coverage is not a proxy for useful tests. But the lack of any significant degree of code coverage does tell you something. And encountering zero code coverage is certainly a problem. So, there is a continuum here, and a balance to be found.

I have come to realize there are many reasons we test, and they are often in competition with each other for getting developer's time to work on them. (Yes, like the Selfish Gene, having us write more code is simply the way tests get us to write more tests, which is really the end goal of it all.)

Lately I have been toying with the idea of creating different suites of tests for each of these testing concerns, so I could track coverage across them independently. I'm thinking of a) basic proof-of-execution tests, which some people think don't tell you much, but in certain codebases, simply validating that you can build a string of SQL from a varied bunch of parameters and execute it without getting a parsing error provides a significant amount of value, regardless of the results returned. And, you can write that test knowing nothing of the business logic. Another level of tests might be concerned with b) the structure of the method: testing all branches or combinations of parameters. Another suite c) would focus on functional tests. And then you have d) mutation and e) characterization tests. For different parts of the codebase, you might choose to favour increasing coverage for one of these suites over the others. You also might wish to add a suite of security or performance tests.

You don't want to get too granular, but the reason code coverage as a metric is not super helpful is we don't necessarily know what those tests are doing, so the more we can constrain their focus, the more meaningful the code coverage results become. And as a bonus, some of these types of tests are very quick to write (a, d, and e), so you can get quickly get to some useful degree of coverage even on a code base you don't understand, or that has no coverage at all. The value of this is not to be discounted, as writing tests can be very time-consuming, and seem like an impossible task when faced with a large, legacy codebase.

Now, I've never worked at a big company with lots of developers, but testing seems dramatically overvalued. My company currently employs zero testing (meaning zero automated tests). Anything that could negatively impact the company if it blew up is examined pretty closely and then set loose. Occasionally things break.

Our company serves tens of millions of users monthly. We have bugs, we fix them as needed. We're not writing banking software so the risk profile is pretty small, but I think more companies fall into this category than test engineers would have you believe.

I'd give myself 2 more years in the industry if I had to do TDD, or anything else that strives for even moderate coverage.

I am a huge fan of unit testing and the best thing about TDD is getting your function signatures and other APIs sorted out before writing your implementation. Having to write the calling function first helps ensure that the called function has a signature that will be useful to other callers, instead of something awkward that seemed OK when you started writing the implementation but later turned out to be imperfect.

Also, TDD wards off the problem of people who write untestable code. A lot of functions simply cannot be tested. TDD eliminates that problem.

The notion that you can get software right the first time seems... a bit naive to me, regardless of the methodology, respectfully. We solved the internal API problems in our app simply by either versioning internal services/APIs (ie. MyServiceClassV2), adding a new method signature and deprecating the old one, or updating the existing method signature, which is pretty safe in strictly typed languages. (And if you write code that uses reflection, you get to be on call when it breaks.)

I find your comments quite interesting. How do you validate MyServiceClassV2 if you didn't bother writing any tests for MyServiceClass? One of the benefits of testing is it enables refactoring. Without tests, code bases just become increasingly haunted graveyards where nobody is willing to change anything.

It's not really about validation. Either MyServiceClass works or it doesn't (whether that's DX, or some other problem), and if it doesn't you make a new one or fix it. Most code isn't so complicated that coming up with the correct method signature is prohibitively difficult without writing tests.

If down the line the way it was written isn't working, maybe you need a V2. Anything important that needs updating to V2 gets updated, anything not worth the effort stays on V1. Usually that's not an issue. Sometimes you have to drop V1 for one reason or another, and that can result in some work, but I reckon it's time saved vs. writing test suites, and it always feels worthwhile to do, vs. writing insurance code.

> Without tests, code bases just become increasingly haunted graveyards where nobody is willing to change anything.

People will be fine with changing things if the cultural standard is that bugs can happen but just try to avoid them. As long as you have good logging it's largely a non-issue. I'd be more hesitant to change code if I had to update 10 tests along the way. Maybe that's laziness but it also feels wildly unproductive.

It really depends on what your building, how quickly you need to build it, and what the client cares about. It could be no-one cares if MyServiceClass has a small bug that causes it to crash and restart from time to time - I worked in a startup that had something like that and it was fine within the needs of the company. Nowadays I work on financial stuff and a crash could cost ££££ so it's unacceptable, meaning we need more testing to check stuff.

Also with interfaces, they matter more if your working in a large distributed team. Sometimes you need to have the method signatures designed upfront so other teams can work on it, meaning it's less trivial than say an internal subroutine used in a private class.

I don't like TDD. A lot of times I don't know what the code will look like until I write it, and whatever's on the screen is just a scratchpad for my thoughts until the final form becomes clear to me.

That said, I love having a good test suite. I updated a major internal project from Python 2 to 3 and my metric for being done was whether all the tests passed. When they did, we launched it in production, and it worked without serious bugs from the very beginning. I can't even imagine what that process would've looked like without some reassurance that I was catching at least most of the corner cases we'd previously identified.

Also, I love tests as a way of asserting that a bug or security issue stays fixed. In fact, I doubt it's possible to pass something like a SOC 2 audit without it.

The problem is that you're missing guard rails. People know they have to be paranoid, so they move slowly; just like people drive more carefully without seatbelts and ride bicycles more carefully without helmets. Care is good, but does not come without cost.

I think it's hard to measure the impact of not testing, because in order to compare A vs B, you'll have to write tests (and it's a large job to write a test suite for existing code), but that's why it's insidious -- you know writing tests is going to slow you down, and you don't know how much time it's going to save in the long run. You end up with a heavy bias for the status quo. And, if f the tests aren't any good, you'll spend a lot of time writing them and you'll just get friction on future changes, slowing you down even more. But if the tests are good, then people that are new to the codebase can arrive with confidence.

Manual testing is good, but it scales O(n) with the number of features. What works for the simple prototype with two big features quickly becomes a drag when every engineer has to test all 100 features against every change they make. (That's why organizations make some other team do the tests, and then the bug comes back to you 3 weeks later, and then you stop what you're working on to fix the bug you caused -- context switching off of new feature work, and delaying that feature. Slow, and annoying!) The solution to this slowdown is usually more process (if we just write it down, it feels like it's not work) and "we need more engineers". The associated O(1) overhead of new proces plus the O(n!) communication slowdown means that the n in n features to test for every change grows much more slowly -- it's feels under control, but what you're really doing is less work with more resources.

I look at the tricky edge cases in my own work, and look at where the bugs creep in (we do a quarterly review of these), and it's always in the "that's too hard to test" code. Some examples: assuming that the test suite's view of the database is the same as a database without the new migrations applied, simple refactoring leading to null pointer dereferences in an edge case, third party applications that call into the API that "has no users" (according to grep), and things like that. These are the things that burn new team members, and make them overly cautious forever. Caution and velocity are incompatible, so to me, it's crucial that these dark corners get addressed, so a passing test suite means a codebase with only bugs we haven't seen before. (There are always going to be bugs, but you shouldn't fix the same bug twice.)

(Oh, and there are definitely bugs in code with 100% coverage. 100% coverage just means you found the bugs you already thought of, but it doesn't mean you found every bug. A test suite will never ensure your code is bug free.)

There are a lot of open source projects that lose their primary maintainer, and they never get another feature again because of this. Someone wants to add one, but they can't figure out how, and just rewrite it, or give up completely. Be on the lookout!

I've always wondered about building a test-sensitivity metric to complement coverage (and thus make coverage more useful):

- Pick a random line of code (skip comments and whitespace if you can)

- Delete/modify it in some way

- Run the test suite and see if it passes

Do this thousands of times on a codebase, then report on the test suite failure rate accross all of the random changes, broken down by file etc.

You could track it all in a long running database and focus each test on lines which have recently changed to make it less expensive.

I've never actually done this, but I'd guess that you could get a good sense of which files have good tests, and (combined with traditional coverage data) which files have bad tests vs just no tests.

This is known as mutation testing[1].

[1] https://en.m.wikipedia.org/wiki/Mutation_testing

Some especially corectness-concerned or compatibility-concerned people use mutation testing, which changes each line of code and fails the test run if no tests fail as a result of a change.

Coverage is:

- a negative signal (“low” coverage = no coverage)

- not a positive signal (“high” coverage only means code paths were reached)

- doesn’t mean your tests are thorough

- doesn’t mean your tests are correct

- most importantly for my current work, doesn’t mean you even have an accurate number (istanbul reports 100% coverage if it doesn’t instrument anything; nested languages aren’t instrumented at all, good luck finding out coverage of an XSL transform in JS)

- (still relevant to my work) a good way of conveying safety of small, methodical refactors that can be eyeballed along with incremental changes to see they’re logically equivalent

Testing is becoming less and less helpful in the "Enterprise" world. I don't even think it's possible in many cases.

The typical CRUD apps out here in the contracting world - many in serious disrepair after a few different offshore or junior teams have worked on them - have been designed using way too many microservices and databases, then duct taped together with message queues and "Cloud" infrastructure like SNS or Lambda.

Unit tests are pointless since 90% of your code does not fit the typical use case they'll teach you about in Bootcamp or in some TDD book - it's not like you're implementing some linked list implementation where you can write a nice, self-contained test suite that goes through a few valid inputs, edge cases, and proves things work.

Instead, you're just assembling dozens of frameworks and managed services and 3rd party libraries, and wrapping it all up with a UI on the front.

Integration tests used to be helpful. But nowadays, you'd need to recreate some huge Docker environment for each test, complete with several pre-loaded databases, half a dozen supporting microservices, and ensure you can connect to a complete test environment in the cloud for your managed infrastructure.

Or you can do what's becoming increasingly common in the Java / Spring world where you have this elaborate system of mocks. Eventually, your mock classes will become more complex and elaborate than the original class itself. You'll spend 50% of your time trying to keep this integration environment working, with all the race conditions it entails as databases, queues, services sometimes start "out of order", and it won't even provide any benefit, because your mocks don't behave at all like the non-mocks do outside of tests.

And don't get me started on front-end testing these days. What I see most amateur developers doing is just instantiating their little React components in some pseudo-browser environment and then testing that, yes, some internal class used by the component library is, in fact, also applied on their test component. Which means they're not testing anything relevant at all. And it also means all the tests break when we move to the next version of the component library, and all those internal classes are renamed or deprecated.

"Next, they selected random subsets of the project's original test suite of varying sizes, ran each against the buggy mutants, and counted how often the subsetted test suite caught the bug."

I'm sorry what? Of course if you only run a random subset of tests, it won't matter whether the test suite achieves 100% line coverage. What a garbage study.

Nobody is arguing that usong a subset of tests won't catch all bugs.

They are stating that having X% code coverage does not imply that you will be able to catch X% of bugs (at least, of the trivial type they introduced here) so you shouldn't use code coverage as a proxy for how good your test suite is.

This may be obvious to you, but I've definitely seen teams fall into the trap of using code coverage as a primary measure of how good testing is.

Say we have an application covered by 1000 tests. We fast forward a few years and (amuse this crazy notion) the application is decommissioned. We look and see that 500 tests never failed; those 500 always passed. Did we waste dev time by writing those tests?

It's an interesting question to think about.

If I have a phone I need for work and insure it because if it breaks I could lose my client, but never actually need to claim on it, did I waste my money?

No, but I don't see the connection to my question. Maybe a slightly closer example is, if I have a phone, and 2 backup phones, and I never use the 2 backup phones, were they a waste?

It's also interesting to see the response from people when I even pose this question, or any other question that might even remotely cast doubt on the CI/TDD dogma. Another question is: how do we rectify the complexity added to the total body of our source code when we add in really complex tests? Is there a point at which we are adding more complexity and increasing the potential bug surface area?

This is yet another paper where the title exaggerates the importance of the conclusion.

I found a copy of the original paper - there are several risks to the model that the authors mentioned:

- They only mutated conditionals to test coverage, not any of the other possible errors the test suite may be looking for.

- Equivalent mutations may be miscounted, particularly when developers test a lot of off by one errors.

- There data may not meet the assumptions of the Kendell r correlation used.

- Most of their data had low levels of coverage - previous research shows high coverage is needed before it is related to effectiveness.

- They did not account for object oriented code boilerplate code (getter/setters) which do not need to be tested, causing their counts to be potentially be off. This is major, as they were using only Java projects.

- They had very narrow inclusion criteria, so the results may not be generalizable to all codebases. The projects had to have over 1000 test written; the average LOC was generally in the 100k range.

This honestly sounds like some grad students final project, not advice for real world digestion.

Goodhart-Strathern's Law: When a measure becomes a target, it ceases to be a good measure

That is pretty obvious.

The main reason is that what we would really want is measure coverage of the specification of the unit.

Now, that would be difficult, so we measure something else with obvious poor result.

Hmmm....how to count tests? By the number of methods or the number of assertions?

Ideally, according to best practices, these should be roughly the same!

"(F)ive large open source Java projects" is hardly a big and diverse enough sample to be making such a sweeping conclusion.

Use the highlighted source code coverage report. It's especially good if the language calls out uncovered branches.

(2014)

Also, the actual paper is paywalled :(

I want my 90 seconds back.

Reply on news.ycombinator.com