Hacker News Reader

Comment by tootie | Hacker News Reader

I work in web consulting where we frequently take on a project from scratch (greenfield or otherwise) and ramp it up quickly in 6-18 months then hand off to the client who will likely maintain it for a year or two or three then redesign (nature of the projects, not a result of quality). My company is pushing for more and more sophisticated automated testing at the time that I'm starting to think that automated testing is costing us more effort than it's actually saving. Our bugs are, as you say, predominantly misunderstood requirements. The tests never catch them. Even with correctly understood requirements, unit tests frequently miss edge cases. Something that gets missed in so many of these process debates is that different kinds of work require different processes. If your job is to maintain a massive software product, then automated tests are probably your bread and butter. But for my line of work, they just ain't.

View on HN

You folks are in an odd situation: you always see the costs of testing, but you never experience much of the benefits. Your intuition of the right amount of coverage is guaranteed to be off.

For me the right amount of automated testing is all about exploring the cost vs benefit boundary, which shifts with tooling and circumstance. But my firm rule now is: code must either have a firm expiration date or full test coverage. Nothing between is allowed.

So if you're going to cut back on testing, make sure your customers have agreed in writing on a date after which they should throw the code away.

When you say "misunderstood requirements" my read on that is a failure of functional testing. Unit testing is good at testing small chunks of code for things like valid inputs/outputs, but not for validating whether the software performs as specified. Functional testing is hard to automate initially, but can be automated for regression testing. Performance testing, on the other hand, is easy to automate. Sounds like your organization is pushing automation as a way to control costs, without perhaps fully understanding what can be automated and when.

Software is not just development, there is also a maintenance perspective. If I have a bug report in a complex piece of code it's a lot easier for me to write a test that covers the use case and work from there then building, fireing up a test device, etc.

Or when refactoring a piece of code that renders UI, write some tests to verify a couple of common cases and then refactor. Run the tests and I'll know whether the output is the same.

I'm saying there's a curve measuring the time spent writing tests and the time the lifespan of the application. Think of like the rent vs buy debate. Tests are a the big downpayment you make hoping that it will pay off in the future. The kind of project my company does are rentals. Same reason we rarely do major refactors.

It sounds to me like the benefit is to the customer, easing their maintenance. Maybe you should raise your prices?

Reply on news.ycombinator.com