Comment by mannykannot | Hacker News Reader

Comment by mannykannot | Hacker News Reader

There are a few relevant facts that should be known to everyone (including managers) involved in software development, but which probably are not:

1) 100% path coverage is not even close to exhaustively checking the full set of states and state transitions of any usefully large program.

2) If, furthermore, you have concurrency, the possible interleavings of thread execution blow up the already-huge number of cases from 1) to the point where the latter look tiny in comparison.

3) From 1) and 2), it is completely infeasible to exhaustively test a system of any significant size.

The corollary of 3) is that you cannot avoid being selective about what you test for, so the question becomes, do you want that decision to be an informed one, or will you allow it to be decided by default, as a consequence of your choice to aim for a specific percentage of path coverage?

For example, there are likely to many things that could be unit-tested for, but which could be ruled out as possibilities by tests at a higher level of abstraction. In that case, time spent on the unit tests could probably be better spent elsewhere, especially if (as with some examples from the article) a bug is not likely.

100% path coverage is one of those measures that are superficially attractive for their apparent objectivity and relative ease of measuring, but which don't actually tell you as much as they seem to. Additionally, in this case, the 100% part could be mistaken for a meaningful guarantee of something worthwhile.

Partly related: Is equivalence partitioning [1] used much in the industry in software testing?

[1] https://en.wikipedia.org/wiki/Equivalence_partitioning

I came across the concept some years ago, IIRC, in the classic book The Art of Software Testing by Glenford Myers [2], and used it in a few projects. But have not really heard or read of it being used much, from people or on forums.

[2] https://en.wikipedia.org/wiki/Glenford_Myers

I think it can be complementary to some of the other approaches mentioned in this thread, which, BTW, is interesting.

In the form discussed in [1], it gets pretty complicated when more than one operand is a variable, and they are derived from prior operations on the original data.

Alternatively, identifying equivalence partitions from the semantics of the input data faces the problem that a faulty program may create invalid partitions, so some test cases for a given equivalence partition pass, while others fail.

You raise some interesting points about issues with the method.

I had used the approach in your second paragraph above.

I agree that it can have the issue you mention. But is this not more or less the same as the issue that even test code can have bugs in it? But we still use test code. For that matter, even human testers doing manual testing can make mistakes. But we still do manual testing.

I agree that it is a valid technique, and in fact it is widely used: a minimal case is when we use a test account ID to stand for all accounts. The pessimist in me was looking for the exceptions, which is actually not a bad trait when you are testing.

> The pessimist in me was looking for the exceptions, which is actually not a bad trait when you are testing.

Agreed :) Good discussion.

Also: the fundamental undecidability of the Halting Problem

If you actually know what the halting problem is, you know that if your code triggers that, it is pretty much broken. If you're not writing a programming language.

... is really not relevant to software testing.

Reply on news.ycombinator.com