> Failing tests indicate the presence of bugs, but passing tests do not promise their absence.
If only :)
Far too often I find myself working with tests that patch one too many implementation details, putting me in a refactoring pickle
If only :)
Far too often I find myself working with tests that patch one too many implementation details, putting me in a refactoring pickle
I’ve written plenty of do nothing tests in my time to be sure that management regularly got a report of tests being added.
- test that were useless, are still useless and will always be useless
- tests that are currently useless but were used in the "wtf should i write" phase of coding (templating/TDD/ whatever you want to call it).
I'm partial towards the seconds, and i like when they're not removed, because often you understand how the API/algorithm was coded thanks to them (and its often unit tests). But ideally, both should be out of a codebase.
But look! Thousands of tests and they all pass! Taste the quality!
I've also seen plenty of tests that test if a template was rendered rather than if whatever thing it actually outputs was in the output. It is just calcifying the impementation making it hard to test.
But it is a tradeoff, and a hard one as well, because if you do all things all the time, combining all variations of database with all variations of the views, then you end up with a test suite that take forever to run. Finding the right tradeoff there has not shown itself to be very obvious, sadly.
When done with that phase and my API looks relatively functional, I remove all relatively trivial tests and I write bigger ones, often randomized and property-based.
This works decently well and you do not have an army of useless tests there hanging after the process is done.
Been there. Change one tiny thing, and 20 tests fail all over the place. But hey, at least we had ~95% test coverage! /s
The more time some piece of code has survived in production, the more "trusted" it becomes, approaching but never reaching 100% "trust" (I can't think of a more precise word at the moment).
For tests it's similar; the longer they had remained unchanged while also proving useful (e.g. catching stuff before a merge), the more trusted they become.
So when any code changes, its "trust level" resets to zero at that point, whether it's runtime code or test code. The only exception might be if the test code reads from a list of inputs and expected outputs, and the only change is adding a new input/output to that list, without modifying the test code itself.
Tests that change too frequently can't be trusted, and chances are those tests are at the wrong level of abstraction.
That's how I see it at least.
It just mean that for 10 years, this codepath has not been taken (The conditions for this specific error case was not met for 10 years) :-(
Actually, it would be a good monitoring information to know which path are "hot" (almost always taken since the beginning), "warm" (from time to time) or "cold" (never executed). It could help build a targetd trust. I guess that it might be possible for VM languages (like based on JVM) because the VM could monitor this... but it might be harder for machine code