When TDD Fails

I got as far as: Oh, and good luck mocking your database and HTTP request objects. and just stopped. If you don't have any experience with TDD, then don't write about it. Mocking HTTP request objects is incredibly trivial and tools are written in every language to do just that.

I don't know why these articles keep getting written. No one's forcing you to write your tests first, and if you can't wrap your head around the benefits (which the author clearly can't) then don't do it. It's fine. I'll still continue to utilize TDD, and continue to pump out significantly more code (with significantly less bugs) than the code I produced without TDD. Yes, there's definitely code I write where I don't write the tests first. It happens, sometimes it's just easier to bang something out because you aren't sure what you're building. That doesn't mean TDD is a failure, it's a tool, like anything else. Use it where it's appropriate, because the benefits are massive.

I'm currently building a test framework for my graphics engine, and have run into the exact problems described in this post. A graphics engine, by design, must be able to do millions of different things that are created by the interaction between relatively few methods, none of which can go wrong at any time. One of my new features is probably buggy, but attempting to brute force test border conditions requires tens of thousands of tests because of all the interacting elements, any two of which could interact in just the wrong way to blow something up. It gets even more ridiculous when you have precision bugs, where only certain numbers in certain ranges in certain cases will explode. Testing that using inputs is impossible.

This occurs so often and with such regularity I am now convinced that everything I write is riddled with bugs that I will probably never find without beta-testing in real-world scenarios, plus many more bugs I will simply never find because they never come up.

A much better design would be a test platform that analyzes the assembly structure to pinpoint actual edge cases in the code itself, which could then be used as a guide for finding bugs instead of relying on hundreds of thousands of test cases.

While testing code is important (and I have tried to do better at writing proper tests for my code), the reason that I have not adopted TDD is because when I write a test and it fails, about half the time it's an actual bug in the code, and half the time it's a bug in the test. I view testing as more akin to going back and checking your work after solving a math problem - not as the definition of what your code is supposed to do, but as a verification that you did it right (and that it continues to work right in the future).

I generally agree with the final sentence: "If you choose a methodology without comparing it against alternatives, if you claim that it works all the time without evaluating the results, than you're not doing engineering - you're practicing a religion."

I've definitely seen some negative effects when a team is forced to create a huge volume of low-level tests because they perceive that to be the only acceptable solution. They get bored with the work, and worse, think less about the larger integration issues.

I'm not arguing that TDD fails, but you'd better monitor the efficacy of whatever testing regimen you employ lest you suffer process and quality rot.

P.S. Dude needs to make friends with a spell checker... wow.

IMHO, good testing is hard. I think that something that's probably as hard as writing programs in the first place shouldn't be commoditized into a methodology. I don't particularly dislike TDD, but I certainly don't like it either.

The best phase to write tests is when you've locked down a part of your program. A part such as one distinct submodule or function or the sort.c or nodegraph.c of your latest project--a part that is relatively orthogonal to the rest of your code. That sort of ensures that the basic blocks, once finished, won't fail surprisingly. However, this can only be applied to basic building blocks.

Testing the bigger parts of your non-trivial medium-size program is likely to be so hard and complex that you have no chance at planning testing beforehand. I think that a good programmer or tester can come up with a relatively comprehensive testing suite that triggers execution paths up to 80-90% code coverage if given a sufficiently finished program, i.e. the program structure has mostly stabilized. Good programmer can also make changes to the same program without decreasing the quality of code. Bad programmers are as able to write good testing as modifying the code itself without letting entropy to the driver's seat.

The author argues that TTD fails in code that's largely wiring. I think the opposite is true. I'm writing an application that's largely wiring, and for the first time in a long time, using TDD. It's refreshing.

Most of my tests are two-liners, like:

    it "should do x when y" do
        obj_under_test.should_receive(:consequence_method).with(some_args)
        obj_under_test.do_something
    end

There are a number of reasons why this has helped:

1. It ensures I'm using dependency injection, etc to write testable, well-factored code. There's huge correlation between testability and maintainability. 2. I don't have to boot the thing all the time to confirm that I didn't mess up in some obvious way. Covering the code paths prevents typos. 3. My test suite runs in under three seconds. I can sanity check what I'm doing, without being tempted to browse HN/reddit/twitter/etc.

I like TDD more in wiring-only code. If I'm writing wiring, I know what my test case will look like ahead of time, and I'll write it first. If I'm writing experimental algorithms, I have no idea what will happen, and I'd rather write code and poke it.

I've actually come to this conclusion on several of my Rails projects. After struggling with mocking up yet another set of I/O objects I realized that it's not really doing anything useful. I agree that there should be testing, its just that TDD on MVC is very difficult to do properly. :/

The real "problems" with TDD are

(a) in the "driven" part, not the "test" part. Tests are (in general) a good thing. However, using a series of tests to drive your design (aka "TDD is not a testing method,it is a design method" idea) often gives you an illusion of progress as you hill climb using conformance to an increasing number of tests as a progress heuristic and end up on top of a local maximum (as for example in the TDD sudoku episode).

(b)in conflating TDD with one or more of (1) testing, (2) automated testing (3) automated regression test suites (4) developers adding more tests to the automated regression test suite as they develop more features, refactor, debug etc.

You can have (1) to (4) without either (5)writing tests first (aka "don't write a line of code without having written a test covering it") or (6)driving your design with tests. The last two ideas are the real distinguishing features of TDD and are of debatable merit. None of (1) through (4) are novel ideas. (5) and (6) are where differences of opinion happen.

Even if you choose to use TDD, it is good to be aware it is just one tool in your toolbox and not necessarily the default tool to reach for.

(c) in the zealotry of some of its evangelists who insist that TDD is some kind of moral imperative and is the only "correct" way of developing software and anyone who doesn't follow that path or make respectful obeisance to it is "unprofessional","dodgy" etc. This is often accompanied by conflating TDD with more generic notions like "automated tests" etc as above.

For example, Rich Hickey, the author of Clojure, said recently at the Strange Loop conference "We say, “I can make a change because I have tests.” Who does that? Who drives their car around banging into the guard rails!?"

(and that is all he said. One sentence in a keynote presentation)

For this Hickey was taken to task by a TDD advocate, Brian Marick, for not being "respectful" enough to TDD and for his "tone" in daring to mildly criticize it as a development practice. After some tweets complaining about Rich Hickey's tone driving away people from Clojure etc he wrote

http://www.exampler.com/blog/2011/09/29/my-clojure-problem

"The dodgy attitudes come from the Clojure core, especially Rich Hickey himself, I’m sad to say."

This kind of repeated whining and harassment over a few days made the normally unflappable Hickey (who asked for references to his "disrespect" etc, to the sound of crickets) lose his temper and say (on his twitter stream)

"If people get offended when their tools/methods are criticized as being inadequate for certain purposes, they need to relax.",

and "Testing is not a strawman. It's an activity, it has benefits, but is not a panacea. No 'community' should be threatened by that fact"

and later "Accusing people who merely disagree with you of being snarky, intolerant, dismissive etc is both wrong and destructive."

and much later after being subjected to a barrage of tweets criticizing his tone and 'lack of respect' for TDD, "If launching an ad hominem attack is the product of a lot of thought, it is time for you to move on. Good riddance."

postscript: the best criticism of TDD I've seen is at http://www.dalkescientific.com/writings/diary/archive/2009/1... . The responses at http://dalkescientific.blogspot.com/2009/12/problems-with-td... are (mildly) interesting as well.

No one at my work place, including me, understand unit testing or TDD. I was recently asked to add a test suite to a service I wrote that is basically a simple wrapper that returns the result of a stored procedure call. The only test I could think of was to call it with all null parameters, in which case there should be no output. Other than that, the results depend on the state of the database. I'm familiar with the concept of dependency injection, but I couldn't add it to this very simple service in good conscience, since I knew that adding the necessary complexity would only increase the likelihood of a defect.

The author is complaining that when requirements change that his tests need to be updated/rewritten.

Is he joking? I mean if the updated requirement changes the behavior of the code then the test better freaking fail and require the test to be updated, otherwise the test (if it even exists) is terrible.

I very rarely use TDD, but I am a fan of it. First off, absolutes like "always write tests" are for people that are bad at programming but are still employed as programmers and can't be fired. They haven't developed the judgement for when to write a test or when not to, so in the interest of getting some reasonably useful test suite, you say "you must write a test for everything".

Secondly, I don't really agree that these action methods are untestable. Sure, "print hello world" is untestable, because it's so simple that you're not going to fuck it up, and because there's only one execution path that can possibly occur. But most methods are not like this; they need to reject invalid data or state, they need to craft database queries, and so on. In that case, you very well can write good tests for this sort of thing.

Say I have some code that needs to accept an HTTP request that has a "foo" parameter:

    def do_something(self, context, foo=None):
        if foo is None:
            raise context.UserError( 400, 'You must supply foo.' )

        context.get_model('Something').apply_foo_to( user=context.current_user(), foo=foo)
        return context.get_view('Something').render()

This is easy and valuable to test:

    container = DependencyInjectionThingie()
    context = container.get_fake_context( current_user='jrockway' )
    controller = container.get_fake_controller( DoSomethingController )
    something  = container.get_fake_model( Something )

    # ensure that empty foo is rejected
    raises_ok( lambda: controller.do_something( context ), UserError )

    # ensure that something is mutated correctly
    controller.do_something( context, foo='OH HAI' )
    compare( something.applied_foo, '==', ('jrockway', 'OH HAI' )

(Who would have predicted the day where I started writing my HN examples in Python!)

In just a few lines of code, we get a little bit of extra security around our do_something action. We are sure that a UserError is thrown when foo is not provided (which our imaginary framework turns into a friendly error message for the person at the other end of the HTTP connection), and we're sure that the model is mutated correctly when foo is valid. In three lines of code.

I find that people that have the hardest time writing tests have poorly-architected applications that don't lend themselves to easy testing. The key point to remember is: if you don't pass something to an instance's constructor or to a method, don't touch it. Then everything is easy to test, because you can isolate your function (or class) from the rest of the application, and then test only what that function is supposed to do. (In this case, the fact that a UserError exception becomes an error screen is something you test in your framework's tests. Same goes for the fact that view.render() renders a web page; test that in your view tests.)

This style of development is also good for more than just testing. A few months ago, I wrote an application that monitored a network service. Not wanting to rewrite the service or mock it, I pointed my tests against a dev instance of this service. Everything was great until the service blew up on a Friday night and nobody was around to fix it. Faced with not being able to write any more code until Monday morning, I knew I had to fake that service somehow. 20 minutes later, I had a class with the same API as my "connect to that service" class. I changed one line of code in my dependency injection container (to create an instance of my connection-to-fake-in-memory-server instead of connection-to-networked-dev-server), and then I was back in business. That's the beauty of writing code to be flexible: you don't have to get everything right on the first day.

(People will argue that tests should never depend on external services, because they can blow up and then you're fucked. Yes they can, and yes you are! But while I didn't do everything right on the first day, my design allowed me to recover from this mistake without any code changes. And now I just run the test suite twice before a release; once against the fake connection and once against the real server, just to make sure that whatever assumptions I made in the mock server also hold when connected to a real server. I like releasing code that I know works in real life in addition to my fantasy mock world, but that's just me, I guess.)

Edit: and oh yeah, it's easy to mock databases and HTTP requests. We've seen the second one already; you let your framework translate between HTTP and method calls, and you write the tests for that when you write your framework. This frees up your application developers to Not Care about that sort of thing, allowing them to write great tests with minimal effort. The first one is also easy. You write code like:

    class UserModel(Model):
       def __init__(self, database_schema): ...
       def last_login(self, user):
           self.database_schema.get_resultset('User', user=user).related('LoginTime').order_by('time DESC').get_column('time').first()

Then when you're testing your controller, you pass in a fake UserModel that just defines last_login as something like:

    class FakeUserModel(Model):
        def __init__(self, database_schema): pass # don't care
        @memoize
        def last_login(self, user): return datetime.now()

The code to ensure that last_login generates the right sequence of operations on your ORM is somewhere else. The test that your ORM generates the right SQL AST is somewhere else. And the test that tests that ASTs are converted to correct SQLite SQL is somewhere else. You already wrote and tested that code. Assume it works!!!

Yes, sometimes you will write a few end-to-end tests to ensure that when an HTTP request that looks like foo arrives on the socket, you write a HTTP response that looks like bar to that socket and your database's baz table now contains a record for gorch. But that's not how you test every single tiny thing your application does; it takes too long, it's hard to get right, and it buys you nearly nothing.

So I guess I add: testing is hard if you write your tests wrong.

Unit testing is not the only form of testing.

I don't know, it's not that hard.

Write acceptance tests (with Cucumber or something), use a good unit test helper like Shoulda for your standard unit tests, and write unit tests for any complicated methods.

The acceptance tests will cover pretty much everything, and the unit tests will cover anything that's hard to test from a high-level point of view.

This is a difficult topic. I've found that adhering to TDD is just not realistic in some cases.

For example, I'm following the Steve Blank approach of Customer Discovery, building my MVP. If I take the TDD approach (which takes more time to code a finished piece of functionality), and successfully iterate enough times during the Customer Discovery process, I throw out all of the tests that I built and am now moving onto something different that I've discovered a customer is actually willing to pay for.

IMHO, there're two points here; 1. Unit testing is not TDD 2. TDD has nothing to do with software testing

TDD is good for developing rapid changing codebases with short development cycles with known requirements and a harsh time pressure, but it's a front loaded way as well.

By which I mean, you have to invest some hours beforehand to make sure you're not shipping crap because you didn't have time to see what would break or to make sure you didn't skipped the controller your GUI developer needed.

TDD tries to ensure one thing only: "Awareness". You accept the initial costs of TDD if awareness is a big issue for you. Otherwise, use something else which serves you to solve it.

If you're aware of what you will be implementing and if there's a mechanism you can check your code against, it'd be efficient, right? TDD uses unit tests for documenting that "Awareness", aiming to utilize the benefits of unit testing as well.

You're right, that it is not feasible trying to test every aspect of your solution via unit tests and that's why there are integration tests, systems tests and acceptance tests. So if you're planning to find defects via the unit tests you're writing for your TDD cycle, think again. Unit tests are good for checking "completeness" and a great tool helping with regressions. Nothing more and nothing less.

As a good engineering practice, we adopt the method of working, based on how we planned to work. If we find TDD fruitful to implement, we avoid putting any logic to controllers. We implement our business logic behind the controller, which also cleans it off of implementation specific crap. Besides us using a completely detached GUI layer helps us write controller unit tests running via HTTP.

We simplified our working environment and make it suitable for TDD, by ways we found to provide more capabilities to us. If we're not to use TDD, we employ other logics and structures, which would fit well with the method we'd use instead.

Long story short, TDD gives what TDD is intended for and as long as you cooperate. If you expect more than it can provide, you'd be disappointed. Regardless of your development method, you have to make sure your development/architecture models and your tool set complies with your method of choice. The rest relies in the question "What do you need your development model to solve for you?"

All the best

"TDD is nearly useless when your code is the opposite of what I've described above: specific and mostly trivial, with complexity coming from the sheer amount of methods and their interactions. "

In other words, badly factored code is hard to test and maintain. Its just that with TDD its more obvious why badly factored code is bad.

Using TDD everywhere is stupid. I think, tests are most useful between layers of abstraction, especially if you are the provider of abstraction. If you are merely a user of abstraction, it is stupid to unit test that code.

If you have an interface to the outside, then that should be unit tested.

To the dev's who take the time to write tests, when I inherit your code base I feel like buying you a beer!

For the dev's who don't - no matter how 'clean' the code. FFFFUUUUUU

  > Okay, maybe I'm going abit overboard with fake qotes.

In other words, a straw man?

This blog and some of the comments below are just crap. If you haven't TDDed before (for a non trivial project) you simply don't know what you are talking about, and it's apparent in your ignorant opinions.

Sorry if that's a bit harsh but I'm tired of seeing these posts by people who don't get it.

I don't get quantum physics, that doesn't mean I'm going to write an blog "when quantum physics fails"

If you don't understand something... Learn about it, practice it, then criticize its actual flaws.

Don't just get frustrated at something and claim its crap. Because its clear to those of us that do understand it, that you are talking crap