That‘s a bold claim. Reeks of hubris though.
Healthy scepticism would convince me a lot more that we can trust your products.
You may also consider if maintaining that test suite for „all cases” is a good investment of your time.
That‘s a bold claim. Reeks of hubris though.
Healthy scepticism would convince me a lot more that we can trust your products.
You may also consider if maintaining that test suite for „all cases” is a good investment of your time.
How can I make sure that I have no leaks?
1. I design software by hand. I design construction and destruction chains beforehand.
2. I implement the modules one by one, create some test suites, plug to valgrind, make sure that it has no leaks.
3. Chain the modules together, re-run the tests.
4. For every component and chain which passes the test, "Seal" the unit. Any change requires whole set of tests again.
For this set of components, since components doesn't change, test suites are also "sealed".
After every build, I have a CI/CD pipeline which runs a series of tests including, unit, module and end to end scenarios. Test the result with ground truth up to 32 significant digits. If something doesn't hold, flag the build and fail. I also keep timing values for certain states, and they shouldn't deviate much. Remember, we need speed.
We should have invalid inputs. However these inputs shouldn't reach to the processing state and just be marked invalid and thrown out. Apply the above pipeline. Implement, test and seal.
Since the stack is stable, and this is an "old school" C++ code, I don't need to migrate libraries and other stuff around much. So, the test suite I've written doesn't need maintenance unless the seal is broken.
However, with the feature I'm implementing, I need to break a couple of these seals, but no biggie. Extend a function, write a couple more unit tests, run the test suite and hammer the code, seal it again. Usual tests will go on, of course.
End to end valgrind tests are done periodically, with not every build. It takes around ~12 hours to complete that with full tracing and reporting.
I'd rather be methodical and give the code I've written a torturous shake down, instead of saying this looks good and move on. I trust myself with the code I write, but not so blindly to go over the top and say that "I'm the one". Instead I do my best, but I believe that I'm the worst coder around here, so I test to break my code. Not to validate.
I still doubt you cover "all cases". Even for a simple problem (e.g. given three integers interpreted as side lengths, decide if they result in a equilateral, isosceles, or scalene triangle), the number of cases will be daunting (in our case: 65, see Robert V. Binder, "Testing Object-Oriented Systems. Models, Patterns and Tools").
Despite the effort you chose to invest into your framework, I still doubt you achieve anything close to this depth within your system and the underlying stack.