I used to think this was a good idea too, until I saw the real statistics on a project on this.
This project (50+ developer team) tracked all bugs, and also if they were regressions or not. Almost 0 regression occurred of bugs that were fixed before.
All testing needs to consider return on investment. The reality, at least for that project, was that testing time was best spent elsewhere.
I personally believe that this advice is one of the most important one actually for a very specific reason: it leaves a very bad impression to client when a bug keep happening again and again after being fixed. Unfortunately I've lived this experience when I was just starting out in my career on a similar project (around 45-50 developers) with basically no tests at all. It wasn't fun explaining to the client, even if they were internal to the company, that the bug we fixed last month had to be fixed again.
Only in the sense that:
- What's this?
- A device that keeps tigers away.
- But there are no tigers in Los Angeles!
- See how effective it is?
>The goal of doing a test for each bug is to prevent them from happening again. Unless you meant that this project didn't apply this particular advice but was getting away with it just fine?
No, he means that they had code to test for the presence of fixed bugs, but nobody reintroduced said bugs and triggered that bug catching test code ever. So even if they didn't have the code, the end result would have been the same.
Perhaps the tests don't run on the devs laptop, but on an integration system (we have such a setup).
(And of course you can make test suites report failures centrally, whether they tests run on a laptop or not).
Integration tests were really heavy, and ran multiprocess on a server. So locally we would run the unit tests, and relevant integration tests. The rest was tested on the integration server, which automatically created bugreports when something failed.
Except that there are, and are kept from terrorizing the population by very effective devices: iron cages. (iirc there is at least one Sumatran tiger in the LA zoo)
Very effective. Especially if you consider that there weren't any ;). You can't get any better ROI of zero effort! ;)
Each project is different of course, and if you saw this in your project, than it would probably make sense to implement them.
But all I'm saying is that this shouldn't be a hard rule. Look at the project, look at the current issues, track your bugs, and draw your own conclusions.
if there is some brain fart, and its tracked down and fixed, the value of encoding that in a unit to be run until heat death. as you say, exactly zero.
but if there is a problem area that keeps throwing bugs like a metastatic tumor - say race conditions or memory footprint, or unicode handling. in that case the return on writing tests to try to shake out those areas is pretty high.
I think the real failure here is to try to substitute informed judgement calls and meaningful discussion with simple checklists of dos and donts.
I agree, but in that case you shouldn't test for that one specific case that failed, but try to make a test that handles similar issues, or tries catch bugs of the same kind.
Of course it depends how far you can push this.
It’s not quite the same though as adding a test to verify a bugfix.
Manually testing your bugs is still cheaper in time and effort than writing unit tests. If it was not, everyone would write unit tests.
Unit tests need to be able to fit into your codebase, sometimes contain bugs too, need to be maintained, slows down your release cycle, etc. I'm not claiming they are bad, I'm claiming it's a trade-off that you make in favor of quality, stability, etc.
Unit tests also need to be produced by expensive, probably scarce programmers. Sometimes it's better to hire 'cheap' testers that can script tests into your framework.
That's why I like to talk about return on investment. If you put in the extra effort of writing a test, you expect this to pay off in the long run.
Nothing is cheaper than not testing your bug after you fixed it. But the chance that a lot of time will be wasted afterwards because it wasn't actually fixed is huge. So basically you have to find the most optimal way to spend your effort.
And like I said previously, our statistics showed that writing unit tests on found bugs wasn't the optimal way to spend our programmers effort.