The evil unit test. | Hacker News Reader

The evil unit test.

makinggoodsoftware.com

77 pointswtfdeveloper52 comments

I've found that the higher level the test (i.e. unit -> integration -> functional) the better they catch things, but the harder it is to figure out what broke.

For instance we had a rule at a place I worked, where we needed to have 80% unit test coverage. And what is described in this article happened. Anytime we'd make a small refactor, we'd have to go update 3-4 test libraries. And if it was a big one, then you have a lot of tests to update. And since some were complex (or since there were a lot of them) people didn't dig into that much. When people finally did it sometimes showed the test wasn't even testing what it was supposed to anymore.

The other issue is we had a lot of bugs around the integration of data. For instance if the DB class returned back an empty array vs null vs empty string. When we did the unit test we mocked it to return what we thought it should, which may not be the correct one. Integration tests better caught this, functional tests even more so.

It seems a lot of shops don't use tests at all, now working at places that have done both, I'd rather error on the side of fewer tests, and do it smart instead of hitting a metric and assuming your code falls into line.

Sounds like your test were quite brittle, if refactoring application code causes tests to break. A good mental rule I use is: "test the what not the how". It also a good reason to program (and test) to interfaces than to the implementation, although you have to be careful with this as it is all too easy to go OTT in your use of design patterns. You have to know how to strike the right balance.

There are various kinds of tests, from unit tests to behavior driven testing. Refactoring application code may cause tests, especially unit tests, to break.

If you're writing unit tests, by definition, you are testing the implementation - that's the point. It seems you're referring to behavioral (or black box) tests, which are very different.

Agreed refactoring method signatures will cause tests to break, but they should be relatively trivial to fix.

Black box tests are not limited to behavioral or system integration testing. you can black box at the unit level. In fact if you are doing proper TDD you will write the tests first, which help to define the interface of the class under test, make sure the test fails to ensure you are testing for something, and then finally write the implementation to make the test pass. Such an approach implies it is black box as you've defined your interface before your implementation.

Remember you are testing what it does, not how it does it.

> Remember you are testing what it does, not how it does it.

While I tend to agree with you (I strongly prefer black-box testing over white-box testing), this is a hotly debated topic and many people feel quite strongly the opposite.

You can Google around and find many advocates for white box unit tests. Martin Fowler's article from several years ago I think does an excellent job of presenting a balanced look at both sides: http://martinfowler.com/articles/mocksArentStubs.html

I've not come across the article before. The section on coupling tests to implementation I guess is the most pertinent part (http://martinfowler.com/articles/mocksArentStubs.html#Coupli...).

> Coupling to the implementation also interferes with refactoring, since implementation changes are much more likely to break tests than with classic testing.

Fowler acknowledges two different styles of tests - state/behaviour also can be termed as classic/mockist. He doesn't really advocate one over the other. I do mock when I have to, but only when there is no other way to setup my test. It is certainly not the default. I find that taking this approach does not make my tests brittle, and makes them robust enough to handle the majority of refactoring exercises.

Of course YMMV.

Yeah that's a good breakdown of the different approaches.

We started using DI, and then had a rule that all unit tests need to be isolated. Which means mocking everything they touch.

For small methods it makes them very fragile, although the other way you end up getting a lot of intertwined tests that can wreak havoc after a lot of tests are made if you're not careful.

I've found that picking what you want to isolate seems to work the best. Our unit tests isolated everything, integration isolated any connections (i.e. DB/Services/etc) which made the tests much less fragile (and much more valuable), but made the errors harder to track down.

Exactly. You want to verify requirements not implementation details.Requirements change less frequently than implementation details.

This just means your tests are bad. Write good tests and you won't have this problem.

Integration and functional tests are good, but only as a quick sanity check. The nitty gritty details need to be handled by good unit tests; the functional tests only tell you things like "did the RPC protocol change" or "does the database cache actually cache something".

It is really easy to write bad tests. That doesn't mean that you shouldn't write any, but just "writing good tests" can be really hard.

For me, unit tests are to assure that if I refactor the code, the places where the code is being used won't be affected by my changes. If I have to rewrite the tests, I've lost this assurance.

There is a time and place for unit tests - when you're creating algorithms.

Most of the time, however, we are creating web apps and things that run at a much higher level of abstraction than the underlying algorithms. We should probably write tests at the same level of abstraction our solution is.

Although, I guess, when you're writing medical software or something for a bank, you should be a bit more strict than when you are making the next best social sharing app.

A guy I know once solved a bug by having a unit test for the md5 function from the standard library of the language at hand ... turns out a specific version was buggy.

Either way, I prefer using integration/functional test. What I care about is that the right output flies in for a certain input. What I don't care about is how. So I don't test for that.

Also, Unit tests must be created before the code they test.

yes they should. It's very helpful to think about how your code will be used before you write it, helps you think it through as well.

"There is a time and place for unit tests... I prefer using integration/functional test."

Personally, I've given up on categorization entirely. It's just "automated tests". People seem to be very prone in getting caught up in the question of whether it's a "unit" or "acceptance" or "integration" test when that's a completely uninteresting question.

Even "Test what, not how" is only a rule of thumb; if I've got a complicated "what", I'll sometimes at least probe the implementation to be sure I'm on the right track during development. Of course, if I change the "what" I just nuke them. I say this having done this entire cycle over the last week. My "what" was too slow and I need a new "what", but unit testing the "what" helped me get it to the point where I could benchmark it much faster than if I had tried to "not test the what". (Sometimes, you can also nuke the "what" tests before putting them into long-term storage.)

If you can avoid getting caught up in attaching goodness and badness to the terms, unit/acceptance/integration can still be useful communication terms, but I resist using them for anything else, and I don't personally use them at all.

(Also, to be clear, I consider this amplification or reflection upon your post, not disagreement.)

Categorization exists so we can communicate. Labelling abstractions enables us to grok more in our head, and communicate faster. Your second paragraph demonstrates this:

"My "what" was too slow and I need a new "what", but unit testing the "what" helped..."

If only there were higher levels of abstractions to say what you mean.

There are higher labels of abstraction to say what I mean. In fact I could describe it in great detail. My boss may not particularly appreciate me using that level of detail. For that matter you probably wouldn't either, it isn't that interesting to the core point here.

My explicit point in this case was about how the labels are actually bad, get in your way, and shouldn't be used because they hurt more than they help.

Usually the problem you'll get if you use a lot of functional/integration testing is that your test suite will run too slowly for you to run it often enough to validate your design. It ends up being something you run when you think you've completed a feature to make sure it works, but not after you've extracted a class to make your code cleaner.

I have also had times when I moved away from unit testing in favor of higher-level tests, but ultimately found that the quality of the code got worse and the build time became unacceptable. I'm not trying to generalize here, I'm only talking about my code on a particular project at a particular point in time.

"Usually the problem you'll get if you use a lot of functional/integration testing is that your test suite will run too slowly..."

I've had good luck attacking this problem directly. If your integration tests are sufficiently integration-y, often this slowness directly reflects the fact that your system is slow to the users as well!

Or, failing that, that you've got too many dependencies hardwired into your system, or some other real problem.

Following up on my other proximal post here, the question is ultimate not whether it's high or low level, it's whether it's fast or slow, period. I've written slow low-level tests because they fully tested a 10x10x10x10x10x10 state space accidentally, I've written (after some work on the underlying system) some very fast tests that literally simulate user input to the system via the web server at the highest possible level.

The real key is that you can't really follow a testing recipe, you always have to be looking out for the cost-benefits. I suspect the other key is that you must come to internalize the idea that automated testing is really important and if that means spending a man-week to improve your testing environment is what you need to do, then do it. It's really OK.

The problem with creating unit tests before the code is that it simply doesn't work for exploratory programming. I often don't know exactly what I intend to do or what the tools are capable of until I'm fairly far into coding the features. Writing unit tests at that early stage would be a complete waste of time.

If you know ahead of time exactly what the inputs and outputs should be then sure, writing unit tests first is a good practice.

Kent Beck talks about this in episode 1 of http://pragprog.com/screencasts/v-kbtdd/test-driven-developm...

The example he works with is writing a driver for Tokyo Tyrant in java. He begins by writing a high level test of what he eventually wants to be able to do. Instead of implementing it though he comments it out and then starts to play around with potential implementation code inside the body of the test. Once he's finished playing he then gradually refactors code from the test body into the implementation until he's got everything he needs and uncomments the test.

It may well be the case that by the time you've finished exploring that you've realised the test you've written isn't very relevant. This is fine, the purpose of the writing the initial test is to guide your exploration in a productive direction. Once you have some implementation code you can always go back and reframe the test according to your new understanding.

I don't understand what you are saying. How can you start coding if you don't even know what you intend to do or what your expected inputs and outputs are? What are you exploring, and how do you know when you've found what you're looking for?

Another simple example from my work of the moment: my goal is to import points from a file (that I'm assured does have points) using a 3rd party file import library. I don't know in advance how the 3rd party library imports points, so the best test I could possibly write ahead of time is a fairly high-level integration test, "Did we manage to import a point?"

All my actual programming work here consists of adding printfs at strategic points to tease out exactly where I should be looking for the point data. Unit tests are completely useless for this process.

Come to think of it, even once I've got it figured out, it's very hard to see how to usefully use unit tests here...

As an example of exploratory development that's very difficult to do test-first, I've been doing a lot of image processing work lately with the objective of improving the legibility of photographs containing text. Trying to find a way to automatically test the output before putting together a solution is a unique problem unto itself, since any given set of input parameters could potentially "solve" the problem for some images but not others.

Sometimes you have to explore the problem domain and available tools for a while before you figure out where you want to go and how to get there. It's sort of like an iterative prototyping methodology.

http://paulgraham.com/arc0.html

I don't use Arc (or any kind of Lisp) but I understand what Paul Graham was trying to achieve.

I've had problems relying heavily on functional/integration tests.

The first problem is data consistency. Both require consistent data in whichever environment they are running. Not an insurmountable goal, but one that often leads to a lot of false failures. These lead to a lack of trust in the test feedback.

The second problem, as others have already pointed out, is that integration tests and functional tests don't tell you what is failing or why. Can I create a user? Nope. Why? 500 error. Time to fire up the debugger...

The third problem is that functional tests do not scale. Each functional test will take orders of magnitude longer to run than integration tests, which take orders of magnitude longer than unit tests. You very quickly end up in the situation where you have more hardware than people or you no longer get fast feedback.

Personally, feedback is the primary driver behind test automation. It is nice to know what I wrote will probably work and it certainly helps simplify the code. However, the biggest pay off comes from the fast and repeatable verification that what I changed has not impacted the code base negatively.

I used to oppose unit testing because I had a good C++ compiler with all warnings turned on and I foolishly thought that was all I needed and that unit testing was only for dynamic languages that did not have compilers.

I'm a big fan now though. If I have written code that's hard to test, I re-factor the code so that it's testable. I've found that un-testable code is generally bad code (it works OK, but is hard to modify later). I also run my unit tests during the build (right after compilation), so I know right away if something fails.

I've come to really like unit testing. I'm less afraid to change things and feel better (in general) about the quality of my code.

I agree that Unit testing is a power full tool. But tools are just tools, and they can be misused. I see this sometimes at companies where they "make" their developers write unit tests, they are just writing them because they have to and this often shows in the quality of the tests. Everything is green in Jenkins so obviously the functional tester did something wrong :)

C++ definitely also needs unit testing - even with untestable code (even when it's bad code, though not always, it may be easier to test it using something like Isolator++, rather than rewrite greenfield from scratch).

So basically: don't test the implementation and don't test the framework.

Do test the API. Test the output.

Good, smart tests should be a pleasure to work with. They should give you the confidence to change the implementation without mercy. They make the feedback cycle faster, bugs more visible, and documentation more thorough.

The only time I've found unit tests to be a pain are when testing asynchronous, event-driven code. The tests aren't any more brittle, but they really need to be supported by good integration tests in order to find the kinds of bugs that can surface in such systems.

I think the point about unit tests is that they should be the default and not the exception. 100% coverage is an asymptotic value. But if you write the test first you can get pretty darn close.

> So basically: don't test the implementation.

I thought that was the point of unit tests...

> Do test the API. Test the output.

... and that this would be integration testing.

Am I drawing my mental lines separating the range of tests in the wrong places?

I can't really say, but if you read The Clean Coder we might get on the same page.

fwiw, what I mean when I say "don't test the implementation," is that your test should not test "how" a function or object performs its task when called.

tldr; unit tests should test units -- the smallest amount of functionality possible

A trivial example might be an object that retrieves some data from a persistent store and prints it out. It might have a "fetch" method whose argument is a key in the data store.

You could easily write a test case that checks when "fetch" is called that the data is written to the standard output stream. It would pass and you could go home after a hard day of work.

But if the requirements change and you need it to write to a socket instead and format the data for a binary stream -- you'll have to modify the "fetch" method and the tests.

If we were clever though we'd realize that when we wrote the test to see that "fetch" output its result to standard out we were actually asking the object to do too much and our test was testing several different units of work that could be broken down.

Ideally our hypothetical object would take a couple of callable objects that handle things like printing data to streams and formatting it. Then our test for fetch only has to test one small "unit" of functionality -- not several units (retrieving data, formatting it, printing it to a stream).

hth

I find TDD to be somewhat redundant when I code Haskell.

The types cover more ground than typical unit tests -- and if you follow a few simple principles (avoid partial functions and some of the worse part of the libraries), you can trust the compiled code more than you trust UT'd code.

Production-quality Haskell does usually employ tests, and those are a joy to write (with QuickCheck), but there's little need to spend time writing tests at the early coding phases.

When doing large refactorings, you can also trust the type system to guide you and find virtually all bugs introduced.

I agree with only one point in the article; the ability to delete tests. Tests of all kinds should be living, breathing code, just like the code in the rest of your application. As such, tests, like production code, can become outdated and obsolete and need to be updated, changed, and sometimes deleted.

However, I do not agree with the idea that tests are evil. If your tests are failing, or breaking regularly, or are hard to write, then your tests are trying to tell you something. Tests are a direct mirror of the state of your code. If your tests are brittle and hard to manage, don't delete the tests, fix your code. If you need to instantiate a tonne of objects in order to make your tests work, fix your methods dependencies and learn to test your methods in isolation of its dependencies.

Any production code needs tests, this is non-negotiable in today's development eco-system. But I get tired of people who do not understand the purpose of tests complaining that tests are evil. If there is a problem with your tests, then your tests are telling you there is a problem with your code. Clean up your code in order to clean up your tests. However, remember that as code changes, your tests will need to change as well. It is okay to edit, delete, change tests to reflect the current state of the application. Just change the tests first to express how you want to interact with a specific bit of code and then change the code, not the other way around.

The thing with unit tests is that they test units. A unit of code might be a simple algorithm inside one function but more often than not the unit is a more complex interaction that is better covered by a smart test on a whole lot higher level.

I prefer fewer tests that try to touch most of the essential features in one go, rather than several low-level tests that test simple if not idiotically limited cases. I prefer fewer tests and high code coverage: by being smart you can do a lot of testing even with a single test if only it touches the strategically most sensitive parts of your program.

Also, most low-level stuff you will test implicitly by the rest of your code. Lots of your higher level stuff would fail to work if your lower level functionality was crappy. Thus, the basic utilities tend to straighten themselves (bug-wise) early on.

Lines of code have a cost and lots of tests drive up your lines. Less code, smarter code. Less tests, smarter tests.

Actually, his 2x3 example is not very good. The fact that 2x3 is 2 three times is the what, not the how; in a perfect world you would just assert that nxm == n + n + ... n m times for all n and all positive m.

This is, coincidentally, where property based testing like QuickCheck comes in.

Edit: Just pretend 2x3 has an asterisk instead of an x :)

(FYI, 2 * 3 is okay; HN preserves * followed by whitespace.)

Ah, thanks. That was probably the issue, because some of my asterisks worked but others didn't. I'll keep it in mind for the future.

Absolutely - even those of us (like me - disclaimer: I work at Typemock) know that there is a time and a place for unit tests and TDD (not the same thing).

In fact, we wrote about this before http://bitly.com/zdyJfl

We're hosting a webinar on Wednesday about different kinds of testing and when a unit test is appropriate and when another kind of test, like an integration test, may be a better choice - http://bitly.com/xeYSYg.

Unit tests have a time and place. There are times in which they are a must if you want to reduce technical debt and test your code - both if you're writing mission critical stuff or even when you're writing the latest social sharing app (remember the Fail Whale? We all hate that!). But, no, you don't need TDD on greenfield code all the time and you shouldn't have to write tests for logic.

The title and content don't correlate. This would be like having an article titled "The Evil Hammer" and then in the article explaining how "Hammer's aren't actually evil, just you shouldn't use them in situations where a saw would be more appropriate."

I think anyone falling into these pitfalls just aren't writing good enough tests, don't dissuade them from furthering their testing skills, promote skillful testing. And change the title.

A lot of these points apply to any sort of coding convention that a group agrees upon, not just unit testing.

Unit tests, acceptance tests, and integration tests are different things. Unit tests test that the implementation behaves as intended, and generally grope around private internals. These change when implementation changes. Acceptance tests test that the public API is implemented correctly. These change when the API/interface change. Integration tests test that the end system implements requirements correctly, and change when requirements change. Most TDD hate comes from not separating these correctly, or not having suitable tools for each type, or from (gasp!) not actually having a clear understanding of requirements, interface, or implementation.

Many people write bad tests. So stop doing that, but don't stop writing tests. The reason why every public method should be tested is because changing how a public methods works is going to break everything using that public method. If you must make a change, fix the test along with the method. It broke to remind you to fix the consumers of your API.

Good tests don't depend on implementation. Your tests should tests interfaces instead of objects and anything implementing the interface should pass the tests. If it doesn't, your interface is wrong or your test is wrong.

I don't feel that this article is relevant for most programmers using newer languages such as Ruby or Python. There's a simple change in the way I use JUnit that I learned from using Ruby (RSpec, Cucumber) about 4-5 years ago: Don't test methods or functions. Instead test expected behavior e.g. a project requirement. Then you can pretty much avoid every problem listed in the article.

I wrote a similar article not too long ago:

http://www.writemoretests.com/2011/09/test-driven-developmen...

>You shall use a foreach every time you need to perform a loop.

Yes

>You will never access your database if you are not using an ORM.

Hell yes.

(Not necessarily disagreeing with his conclusions)

What about while loops? That's the problem with blanket statements: they're very rarely 100% correct.

You shall use a foreach every time you need to perform a loop.

Works pretty well in Python.

Though it doesn't invalidate the point the author tries to make.

Works pretty well in Python

But xrange() in python is just a roundabout way to get back to having a regular for loop.

    for x in xrange(0,100)
    for(x=0, x<100, x++)

So you think while loops are unnecessary in python?

Sounds like a bit of a hyperbole,

  Blanket statements like 100% coverage, unit test per method are not too helpful, and it's important to understand the point where you will start to see diminishing returns with unit tests but that said well written unit tests with ~ 85-95 coverage by line free of conditional logic (use custom asserts) with meaningful test names/descriptions aren't exactly going to put you in league with beezlebub.

there are really nice comments below in this thread on why testing functionality is better than writing unit test cases to test the functions/methods/code. e.g. wr1472: test the what not the how MrEnigma: the higher level the test (i.e. unit -> integration -> functional) the better they catch things Keithnoizu: verify requirements not implementation details

we understand that the unit testing is trying to test the functionality and making sure that the acceptance criteria is also met at the same time. I would like to mention one more point is that we should also try to write unit test cases not just only test the functionality but also try to do a good code coverage. unit test cases are tools for: 1. functionality testing 2. comply with acceptance criteria 3. a tool to help good code coverage 4. be iterative. be backward compatible as long as it could. (anyway to add logic based on version of the code change? need to think on this). yes it makes it little bulky but will work and avoid needing code cleanup and redesigns.

let unit tests help come up with min 80% of code coverage.

Was this a sarcastic post?

What makes you think that?

Reply on news.ycombinator.com