What, more tests are always the best way to improve my product?
yieldthought.com
yieldthought.com
If you are writing a new class or a new method, by all means, write tests for it. Getting tests around all of your currently existing code, however, is often a waste. You need to test the areas you are changing and the areas which are impacted by the change. In any code base, there are hotspots, places where both change frequency and complexity are high. Tackling those first with automated testing often gives you the best ROI.
Remember, you aren't writing automated tests for existing code to find bugs, you're writing them to characterize the current behavior and get a behavioral invariant so that you can refactor and also so that you can add features with the knowledge that you haven't changed the old behavior in unexpected ways.
100% coverage is a feel-good metric. I can easily achieve 100% coverage with bad tests. But I routinely use coverage as my guide to find things that might benefit from further testing, as in "oooh this path is never tested and stuff is more likely to break here".
I try really really hard not to write useless tests-I use tests to drive the development of my app. I need feature x, so I write a test for it. I end up with relatively high coverage because of that. And because I've gotten more disciplined over the years, I'm only implementing the stuff that passes my tests, so I'm less likely now to have things that aren't covered.
So, I view coverage as a guide, not a rule, but it's not a useless metric at all unless you designate an arbitrary percentage.
When I write tests, I don't care (too much) about coverage as about writing good tests. If I'm confident that I've gotten most cases, new features are much easier to add, even if the coverage isn't that great.
Covering a part of the code you know is unlikely to fail isn't a very good use of anyone's time, I think.
I like to measure code coverage for changed code within a rolling time window, say 6 months. If people are writing tests for all they change, you rapidly get close to 100%. This gets past the issue that in many large code bases, you could probably write tests for ten years straight, end up writing lots of tests for code that doesn't change, and still not get close to 100%.
Just because someone says 'I don't unit test' doesn't mean they're not testing their code.
Most code spends 80% of its execution time in 20% of the code. Test that code and fix the outlying bugs as they come up.
Most of the hard bugs crop up where subsystems interact. The bugs inside the subsystems themselves are usually pretty easy to spot and fix.
The goal isn't to produce a 100% tested product that you can have supreme confidence in when you deploy. The goal is to provide a reliable service that people are going to pay you for. It turns out people pay for things that aren't perfect all the time.
Every line of code you write comes with a maintenance tax. This includes your test code. This is a fact.
Unit testing is a tool. Use it wisely and it will help you. Using it for everything is overkill.
Come to terms with the fact that there will be bugs, no matter how much you test.
Every large team (or team that plans to get large) should consider unit tests to help keep their velocity constant.
Carelessly changing things you don't understand and expecting the tests to save you does not count as "productivity." Good design decreases the amount you need to know to safely and productively hack on a system. Tests don't. Tests can save an experienced and knowledgeable person from their inevitable mistakes -- they can tell you something is wrong -- but they can't direct you toward the right solution. Someone who doesn't know what they're doing can hack by trial and error until the tests pass, but at that point their code is still likely to be wrong.
In most applications I've been stuck on, the tests ended up being an impediment to development -- because the design was poor. The tests helped to reveal dependencies, so people who were working on bits of code could easily identify what dependencies they had to fix when they made changes, but that also served to underscore how badly coupled the various components were.
I guess the moral is that too many mediocre wannabe agilistas try to use testing to cover up bad design.
What the article says is certainly true, there are trade-offs to make, but one needs to take into account how effective and maintainable those tests are.
When they go on to over-generalise TDD as the one true 'professional' approach to all development on all projects, it can become a little grating though. Like all things, the right level of test coverage and the methodology used to achieve it, are a matter of trade-offs and the right trade-off can vary a lot depending on the type and lifespan of project, the way requirements are gathered and the project is governed, the business goals, the people involved, etc etc etc.
I can't document this, but many software projects fail or are forced into rewrites and rescues. That never happens when you have high code coverage.
Has any rescue team ever come into a project and gone 'Well, the test suite is in excellent shape, so this shouldn't be that hard'?
That's what I believe, but many products won't have a huge payoff for unit-testing every method. Most of my code coverage is more like automated QA tests through Cucumber / Capybara / Selenium in Ruby.
Having said that, I would still prefer there is some level of test in my codes especially for those crucial processes, but for the others, I leave it to my users to give me feedback which the code won't simply be able to cover.
Covering every edge case isn't necessary. Tests should cover core functionality and then the ongoing addition of regression tests. We hover in the mid to high eighties with our test code coverage and find getting more than that overly time consuming (Python/Django btw).
80/20 rule coming into play?
Disclosure: I am biased towards test driven development. Of course, I also worked, years ago, at a place selling (niche) program development tools. Some value was placed on correctness. YMMV.