Hacker News Reader

> ...trigger alarms when patient's vital signs met certain criteria set by doctor

Most of us aren't working on life or death code like this. My React app doesn't need 100% code coverage but you put it well when you said "There should be just enough to feel safe"

What I really want to drive home is that you should listen to the engineers who are working on the project. Testing is one of those things that is entirely subject to project needs. Like you say, a React app doesn't need 100% coverage, neither does code running a CI, something far out of the serving path, etc. But we shouldn't have a knee-jerk reaction to an engineer saying we should have 100% coverage.

> between 60% and 100% there were many small edge cases that, had they caused a death, I wouldn't have been able to sleep well.

But you're not saying the right words here. 100% doesn't mean you've covered every edge case (although I suppose 60% necessarily means you _haven't_). I can hit 100% without actually asserting anything.

I think it's harmful to talk about 100% without also considering mutation testing, boundary analysis, manual mutation testing...

If the code was open source I would show it to you. There are definitely cases that weren't covered during the initial creation of the service that came up in QA/prod. The entire process was a war story in and of itself. The tests aren't perfect but I have done everything in my power to make sure they cover all of the use cases that appear, samples of bad data from production, samples of bad data I forced to happen in other services, asserting failure cases, and (most importantly) are vigilantly kept up to date. I also, on my own initiative, found people internally to help run a small QA study where they interacted with this system on a daily basis for about 2 months.

The senior engineers were not supportive of other forms of testing in any way (even the user testing). They flat out refused my proposal to attempt to create an integration test suite to cover it's communication to other services as well as other services consuming it's data (this I started over the weekend and was told to proceed no longer).

tl;dr: You're absolutely correct but I still think my testing procedures were better than nothing.

I feel for you. I argued for unit testing for ten years before they were instituted and even then it was half hearted.

>I still think my testing procedures were better than nothing.

I am sure they were, but you also said:

>take the time to understand why they want 100% coverage

which to me indicates that they are under the false impression that this gives some sort of completeness, i.e. "make sure this plastic fence covers the entire stretch of mountain road".

We have a coverage threshold of 90% and it is helpful because it works as a reminder if you try to check in code that isn't covered by tests. I think that pushing that to 100% would add little value but a lot of overhead. Never tried it though so I fully admit that it's just my hunch.

It's our ethical responsibility to know when and where paranoia is worth the marginal time penalty.

Exactly. You are in the tiny minority writing literally life-and-death code, and for which a lot of other common advice given regarding general software development likely does not apply either (like "move fast and break things".) Also, for your type of application I would probably want 100% state space coverage too.

My problem with "100% coverage" is Goodhart's law ( https://en.wikipedia.org/wiki/Goodhart%27s_law ).

Keep adding tests as long as they tell us something new, give us more confidence, document some regression, etc.

Keep adding tests if they're exposing edge cases (e.g. MAX_INT), even if those code paths are already covered.

Keep adding tests if they're asserting useful things about the results, even if those code paths are already covered.

Stop adding tests when you're only trying to make the coverage number increase.

I had a recent project where:

- 0-80% was adding useful tests.

- 80-98% was mostly useless.

- 98-100% found some really interesting bugs and forced refactoring that made testing easier, and was definitely worth it.

Not to take away from your point, just to remind people:

100% coverage still does not mean 100% of all cases tested - it just means that every line has been run with SOME data and every conditional statement was run once in both directions.

Seems like a very specific concern in a very specific line of business. The vast majority of engineers write CRUD apps for business clients that mostly care about whether it completely explodes or not.

It also depends on the service. Some services are critical infrastructure and should have their edge cases tested. Some services provide non essential functions and can get away with less.

I agree that there is no magic number, but 100% coverage is clearly overkill for the average engineering team. In fact, broadly demanding any code coverage percentage is probably over simplifying the issue. It just shouldn't be 0 :)

Really reliable software does full MC/DC testing, but even that doesn't catch all mistakes. SQLite for example does this but there are still bugs discovered occasionally. It also leads to insane amounts of test code compared to the actual application code. That is much too expensive to maintain for most projects.

I'd never thought about this before, but you can definitely flip this around to a pathological case of doing the opposite of striving for 100% coverage.

The problem with 100% coverage that I think of is that the last 30% is often boilerplate code (generated getters/setters in Java and that kind of thing).

But what if an engineer started with the boilerplate and only then progressed to tests that are actually important? You might get to 30% without testing anything useful at all. Then you might test half of the important stuff and hit an arbitrary metric of 70%.

If someone were ruled by the sonarqube score, they might even do this.

Yeah, thinking about the why matters more than the %.

So many problems with "100% coverage". Even if you measure it as branch coverage, it only indicates how much of the code that is already written has been covered. It doesn't tell you how much code that is missing, eg guards against malicious input, null-checks or other out of range arguments. This reason alone should be obvious enough for anyone to understand that the metric is completely bogus.

As others mentioned it also doesn't say anything about state space coverage or input argument space coverage or all combinatorial paths through your code. Only that some line/branch has been hit at least once.

100% code coverage is a breadth-first metric, and generally sacrifices in depth-first testing (especially of the "core loop" as in 80% of the time is spent in 20% of the code is sacrificed).

The decidability/halting problem hints that perfect testing is an impossibility for systems of any complexity, as in not just too big of a big-O... it's flat out not possible on a turing machine of any power.

That doesn't mean give up all testing, but there is a LOT of dogma in testing philosophy.

I agree with others - mission critical systems (nuclear plants, aviation, medical software etc) have quite different standard for quality.

Excellent comment, thanks for sharing. One ought to really think about why tests are needed and what value they add, and not just simply follow some random (thoughtful) article on the internet.

I'm a huge fan of tests that ensure the app doesn't break in a significant way in production. How many tests that is depends on the situation. For your case I would agree that 100% test coverage is correct.