Hacker News Reader

Sounds like your test were quite brittle, if refactoring application code causes tests to break. A good mental rule I use is: "test the what not the how". It also a good reason to program (and test) to interfaces than to the implementation, although you have to be careful with this as it is all too easy to go OTT in your use of design patterns. You have to know how to strike the right balance.

There are various kinds of tests, from unit tests to behavior driven testing. Refactoring application code may cause tests, especially unit tests, to break.

If you're writing unit tests, by definition, you are testing the implementation - that's the point. It seems you're referring to behavioral (or black box) tests, which are very different.

Agreed refactoring method signatures will cause tests to break, but they should be relatively trivial to fix.

Black box tests are not limited to behavioral or system integration testing. you can black box at the unit level. In fact if you are doing proper TDD you will write the tests first, which help to define the interface of the class under test, make sure the test fails to ensure you are testing for something, and then finally write the implementation to make the test pass. Such an approach implies it is black box as you've defined your interface before your implementation.

Remember you are testing what it does, not how it does it.

> Remember you are testing what it does, not how it does it.

While I tend to agree with you (I strongly prefer black-box testing over white-box testing), this is a hotly debated topic and many people feel quite strongly the opposite.

You can Google around and find many advocates for white box unit tests. Martin Fowler's article from several years ago I think does an excellent job of presenting a balanced look at both sides: http://martinfowler.com/articles/mocksArentStubs.html

I've not come across the article before. The section on coupling tests to implementation I guess is the most pertinent part (http://martinfowler.com/articles/mocksArentStubs.html#Coupli...).

> Coupling to the implementation also interferes with refactoring, since implementation changes are much more likely to break tests than with classic testing.

Fowler acknowledges two different styles of tests - state/behaviour also can be termed as classic/mockist. He doesn't really advocate one over the other. I do mock when I have to, but only when there is no other way to setup my test. It is certainly not the default. I find that taking this approach does not make my tests brittle, and makes them robust enough to handle the majority of refactoring exercises.

Of course YMMV.

Yeah that's a good breakdown of the different approaches.

We started using DI, and then had a rule that all unit tests need to be isolated. Which means mocking everything they touch.

For small methods it makes them very fragile, although the other way you end up getting a lot of intertwined tests that can wreak havoc after a lot of tests are made if you're not careful.

I've found that picking what you want to isolate seems to work the best. Our unit tests isolated everything, integration isolated any connections (i.e. DB/Services/etc) which made the tests much less fragile (and much more valuable), but made the errors harder to track down.

Exactly. You want to verify requirements not implementation details.Requirements change less frequently than implementation details.

For instance we had a rule at a place I worked, where we needed to have 80% unit test coverage. And what is described in this article happened. Anytime we'd make a small refactor, we'd have to go update 3-4 test libraries. And if it was a big one, then you have a lot of tests to update. And since some were complex (or since there were a lot of them) people didn't dig into that much. When people finally did it sometimes showed the test wasn't even testing what it was supposed to anymore.

This just means your tests are bad. Write good tests and you won't have this problem.

Integration and functional tests are good, but only as a quick sanity check. The nitty gritty details need to be handled by good unit tests; the functional tests only tell you things like "did the RPC protocol change" or "does the database cache actually cache something".

It is really easy to write bad tests. That doesn't mean that you shouldn't write any, but just "writing good tests" can be really hard.

For me, unit tests are to assure that if I refactor the code, the places where the code is being used won't be affected by my changes. If I have to rewrite the tests, I've lost this assurance.