Cross-Branch Testing
hillelwayne.com
hillelwayne.com
Not if you're writing your unit tests correctly. Thought should go into what you're testing. It might be true that you're going to find an error in 1% of cases, but you shouldn't be shooting random shit into your test. Or in other words you should "constrain" your "random test". I find it difficult to think of a situation where you actually don't know any helpful mathematical properties of your code - since you've obviosuly written the code with an intention to perform some function. There may be good reason for doing this stupid rotations and powers etc. But at the end of the day there was some non-code motviation for writing this code and that is what you should be thinking about when writing your unit tests.
What the cross-branch testing gives you is an assurance that your code is as broken as when you started. Now, think about what is happening here when you test across branches, you have your golden reference model (which might or might not be right) and you have some python packages that you hope are doing a better job than you at constrained random testing. In fact, their odds of hitting that 1% are no better than your unit tests.
I actually think to some extent testing against your previous implementation has value above the unit test - because my experience is that the first person to write the code probably thoroughly manually tested it and didn't convert all those tests into unit tests, and therefore your confidence in the original implementation is higher than revisions, but the lesson to learn from that is to make sure when you first implement something you push as much of your verification into unit tests.
I hear this all the time. “If you just write your tests correctly, you will not have any bugs.”
A statement like this can only be made with a huge misunderstanding of computation and programming. Try telling all the formal verification researchers out there that the perfect set of unit tests can be determined for any program at all. It is not possible. That has been the holy grail of the verification community for 50 years at this point. See this paper from 1975, Towards a Theory of Test Data Selection: https://www.researchgate.net/profile/John-Goodenough/publica....
In 1988, the Category Partition method of test case generation was introduced in this paper: https://www.cc.gatech.edu/~harrold/6340/cs6340_fall2009/Read.... In it, a much bigger sampling of the parameter input space is selected from for test cases, but it can result in thousands of test cases to write. This method has at least one industrial case study where a team applied it on financial software for Freddie Mac with 0 reported defects after release: https://cs.gmu.edu/~offutt/rsrch/papers/calcengine.pdf.
Really, the mother of all testing techniques is the DO-178C certification: https://en.m.wikipedia.org/wiki/DO-178C. This is what companies use when developing avionics software. The highest level of verification requires 100% modified condition / decision coverage: https://en.m.wikipedia.org/wiki/Modified_condition/decision_....
Given that most people don’t know what MC/DC or the category partition method are, I doubt that there are many actually effective test suites out there. Which makes the frequency of these statements pretty confusing.
In fact I think your entire appraoch to this is wrong.
>Given that most people don’t know what MC/DC or the category partition method are, I doubt that there are many actually effective test suites out there.
Given that most people aren't writing software to go into safety critical applications, it doesn't make sense for them to be designing their test suites to that standard. What I'm saying is that this method of testing boils down to hoping that the master is correct (which it probably isn't) and then trusting someone else's tool to provide coverage for you. This is worse than writing unit tests because the tool you've handed responsibility to for coverage doesn't know anything about your device under test and therefore can't direct it's tests to maximize the likelihood of finding errors. At the very least writing a simple model to test against would be more effective because you'd expose bugs in the detail-orientated implementation.
Sometimes this technique catches bugs on prod that are unknowingly fixed with refactoring.
How do you know that your tests are correct? The problem is equivalent.