Is TDD dead?
martinfowler.com
martinfowler.com
TDD, like all of these things - eXtreme Programming, agile, has good points and bad. Listen to what these people have to say, and then take what works best for you.
To me, TDD is about writing my code and my tests at the same time, not test first. As soon as I write just enough code to do something, I'll write a test to see if it actually works. I'll then expand on that test and perhaps add a few more as I refactor that piece of code.
This means I still get the benefits of exploratory programming, unconstrained by test first, but I also get the security of knowing that my changes didn't break anything. It feels very natural to develop this way, as the code base grows, your confidence in its quality grows with it, rather than the inverse (lots of passing tests). Also a fast test suite which runs on file save is addictive, I cannot imagine returning to F5 Driven Development now.
The arguments always end up in line with "You just have to be better/be more confident/etc etc", or in more blunt term, "I'm better programmer than you, that's why TDD works for me but not for you".
This is exactly how I've taken up TDD. I still sometimes feel like I should write the tests first, but (and maybe this is just a matter of discipline) it's much more satisfying to do exactly what you described. The tests still help guide the design of the software, because it'll be immediately visible to me if I'm writing software that's hard to test (and probably - though not always - less correct).
> F5 Driven Development
This made me spit my drink out laughing; it sounds so ridiculous when you say it like that (because it is). You're right on the mark with the addictiveness of a fast, auto-refreshing test suite, I wouldn't want to go back to what feels like the caveman way to do things.
I think one of the very common sticking points about refactoring and testability is that when trying to demonstrate techniques, you have to pick simple examples to isolate the techniques.
When in reality, those simple examples don't need those levels of abstraction or indirection. DHH's gist is a perfect example of demonstrating something that - by itself - doesn't need to be refactored in that manner.
But it completely misses the point. Those refactoring techniques are essential for when your codebase is 100 or 1000 times more complex. (Jim W. mentions this explicitly in his youtube video.) But the problem is you'd never show that complex a codebase in a demonstration, because the mental load in understanding the prerequisites (before getting to the actual refactoring) is too large.
I think another common sticking point is that people talk about refactoring to make a codebase more testable. And that is a worthy aim and all, but making it more testable isn't the end goal. Making it more testable forces you to separate your concerns/layers, and that's the goal. The reason that is the goal is because as your code grows, you still want the product to be easily changeable so it can be quickly responsive to customer's needs. And if you need to switch out your data repository implementation on short notice due to your site being suddenly popular, that'll be easier/faster if that implementation is separated out.
This exactly, I've made this exact point many times over the years. testing is never the goal.
You should read the whole article, but to motivate:
> This post was triggered by Kent's comment about triply-nested mocks. I doubt that he intended to claim that mocking three levels deep is inherent to, or even common in, isolated testing. However, many others have proposed exactly that straw man argument. That argument misrepresents isolated testing to discredit it; it presents deep mocks, which are to be avoided in isolated testing, as being fundamental to it; it's fallacious. It's at the root of the claim that mocking inherently makes tests fragile and refactoring difficult. That's very true of deep mocks, but not very true of mock-based isolation done well, and certainly isn't true of isolation done without mocks.
> In a very real sense, isolated testing done test-first exposes design mistakes before they're made. It translates coupling distributed throughout the module into mock setup centralized in the test, and it does that before the coupling is even written down. With practice, that preemptive design feedback becomes internalized in the programmer, granting some of the benefit even when not writing tests.
Not that Beck or Fowler are advocating anything different but I think DHH is the one with his finger on the pulse of current development trends.
This. This concisely expresses something I've learnt in the last few years but have struggled to write down. Generalising is great until you generalise in the wrong direction, then it creates extra complexity in all the other directions.
You save time and effort by choosing not to make a definite decision on which way to jump with that abstraction.
Now this is a place where functional programming tends to help. Unnecessary FactoryProviderObjectServices just make code unreadable, because they aren't really talking about the task at hand, but about our own internal abstractions, which become useless quickly. Functional decomposition makes throwaway abstractions a whole lot easier to throw away.
In other words, the controllers of typical web apps which AFAICT is where the technique originated and the source for a lot of the introductory examples.
It's not a silver bullet, but what is?
There is? I haven't seen this "debate" take place anywhere other than the Rails community. I work on open-source and I can't imagine releasing a library where its APIs are not unit-tested. That's a different context than the one you mention. And I don't think I'm alone, every library I run into has automated testing.
Our shop is far from a startup though so very few things on this site have meaning for our product.
How can you correlate directly NPS and using TDD? Maybe it is that, for example, the new management/mindset/etc. that moved the dev team to TDD also moved the customer support team to be more responsive, and that's what increased your NPS.
TDD reducing critical bugs hitting prod or stuff like that, I can understand - but NPS seems a bit odd.
I TDD'd for almost 3 years using Rails and this sounds pretty accurate. TDD pushes back on you a lot more when you have no clear underlying and preexisting architecture.
DHH can be quite outspoken at times but I think he's right to have this discussion and to have it in public so we can all learn something.
Let's say you have production code where A will call B, and B will call C. Basic layer separation. It might be a controller calling a service method, which calls a dao.
If you want to test B, then in my mind, that means you want to write a test that would call B (similar to how A does). And then since you don't want C confusing things, you'd create a mock of C (returning a canned result), and assign it to B.
So... how would you test B without mocking C? Are Martin and Ken simply content to let B call C, which would imply that it is an integration test rather than a unit test? Or is there some other design technique they're using?
Think of it like this: How many times have you deployed an app in production, or against production data, only to have it crash and burn because something somewhere in the data wasn't what you expected? No one would really be surprised by it happening, I think most of us have experienced it a time or two in our careers.
Here's the thing: No seasoned developer is going to recommend that you write a migration process, create a DB with purely dev created test data, and if it passes, immediately pass it into production/into the hands of customers. Most experienced dev's would tell you that's dangerous, and you really need to test it against actual production data first.
In this case, the dev created test data is a 'mock', and the production data is 'reality'. In this context, no one would disagree that the dev created test data is not enough in order to ensure proper functioning.
But many people will tell you that a mock object is enough to ensure proper functioning. This is not true, and the investment required in order to make it true tends to be a lot higher than people expect for the same reason that battle tested code tends to be more complicated than most people would expect is necessary to solve the problem.
That doesn't mean that mocking doesn't bring value, but because the cost of making that mock accurately mimic reality, it means you should reserve it for those times where the alternatives tend to be even more prohibitive.
This is why you'll see a lot of recommendations for using mocks to do things like simulate failed connections at specific points in a protocol handshake. Getting the connection to break at the correct time using an actual network is prohibitively difficult. Getting a mock to do it consistently at the right time is a lot easier to do.
These are the sorts of situations that mocks were originally created to solve. The issue is that somewhere along the line people started abusing mocks in situations that did not warrant them either because the cost of actually getting it correct is too high, or because they didn't actually pay the cost resulting in a false sense of security.
I don't think any reasonable developer will tell you that mocks are worthless, but I also don't think any reasonable developer will tell you that you should be mocking a whole lot.
Mocking has real value there because you can control what C returns when testing B. You might write tests for if C returns true, if C returns false, if C returns null, or if C returns some general exception. There, you're testing that B behaves correctly, given all possible responses from C. At that point, you've controlled the interface between B and C, and limited the ways that C can respond. Because of that, you can be sure that if your test for B fails, it's because of a problem with B.
If you instead don't mock C, then an integration test that tests both B and C (since B is actually calling C) becomes a lot more complex - probably more than twice as hard to nail down a bug.
The trick there is that by focusing on the unit test, you've written a clear, simple interface between B and C, when you might not have otherwise. I think that's an example of how mocking can really help, and how it doesn't lead to the nightmare of mocks-calling-mocks that a lot of people seem to criticize. And rather than being only useful in cases like protocol handshakes, it seems frequently useful, such as in wanting to mock a dao layer when testing a business logic layer.
At any rate, that makes it sound like by not mocking, they're merely doing multi-level integration tests. I just would have expected them to be more focused on unit tests, which to my mind requires mocking if you have a multi-layer application.
Jim W. made the same point about "not mocking" in his youtube video though... and in the same breath said that he does, however, mock at the boundaries. Which makes me wonder if this is all just a terminology confusion. Because for instance, if I'm testing a public method in business logic, I'm not going to mock the private methods that it calls in that same business logic layer - that just seems crazy to me. And yet I would still call that a unit test, since it's still limited to one layer, if not to one single-layer-of-abstraction method. I will mock the dao layer in those cases, though, because that strikes me as a completely separate layer/concern. But I don't know, maybe someone else would say that I "don't mock all that often".
I will respond to 1 point you made because I feel it's the point you're not understanding.
> Referring to my earlier example of B calling C, what if B's business logic is complex, but makes no outbound calls except to C, and C (for simplicity's sake) is also complex, but returns only a boolean result?
Theory and practice. In theory, the only thing you need to rely on is the return value. In practice, that code also relies on the runtime characteristics of the code it's calling, and this includes things like unexpected slowness, as well as unexpected exceptions (depending on the technology you're working in).
This is the entire point about a mock being a theory of reality, and getting that right is difficult.
Java/C#/... can be heavily assisted with the right IDE,(test suit generation,mock/stub generation...).So the cost of testing can be light,no matter how big the codebase is.
In JS,or ruby.One actually has to write everything. Therefore,one would be less tempted to write heavily decoupled code with a huge graph of function constructors/classes. While ruby has a lot of sugar for a wide range of things,that's not javascript case.
So in my opinion,the cost of testing and managing test suits is higher in languages such as javascript or ruby,therefore one is tempted to write as little code as possible,and sacrifice decoupling and sometime testability.
In one hangout,DHH comes with a snippet of code,highly decoupled and testable.But does ruby encourage this kind of architecture?because it's more code to test and more testsuit to manage(and more tests to rewrite when refactoring).
So isnt it more a language /tooling question than a TDD/mock not TDD/mock question?
What is good architecture? I subscribe to the idea that we are doing software engineering and as such there are some "generally" understood principles such as SOLID, the Law of Demeter, Cyclomatic Complexity, etc that provide objective measures of "good" architecture (I apply SOLID all the way up the architecture hierarchy not just on classes).
What I've noticed is that TDD results in code that scores well against these measures, while code that scores well is easy to test (i.e. after writing the code).
Therefore, I think the argument that TDD creates bad architecture is false.
About 25 minutes into the talk we get to the crux of DHH's complaint and it is that Hexagon is an alternative to the Active Record (which he created) and the only reason Hexagon exists is to allow TDD. Hexagon requires throwing away the really useful code that is Active Record.
Hexagon appears to be an attempt to introduce sound software engineering practices (SOLID etc) into the Ruby world (with what success I do not know). Active Record and Rails in general is really useful if what you want is what it does, but sometimes its not. The implied claim that Hexagon is a bad architecture is false. The claim that Hexagon only exists to facilitate TDD is false.
"Mocks returning mocks returning mocks"
I use mocks. Fowler and Beck said on the whole they don't use them which genuinely surprised me. They cited examples of code where the test actually enforced implementation rather than purpose. I think that's probably how I wrote tests for the first few years. Code that results in mocks returning mocks returning mocks is code that is violating the Law of Demeter. Its bad code. It happens to be really hard to test, and it happens to be really hard to write tests first that way. Universally I've only ever seen tests like that when the tests were written after the code. TDD doesn't produce code like that because its easier to refactor it rather than keep digging that hole.
Mocks returning mocks returning mocks is a symptom of not doing TDD.
"My mind works differently... I have to write code first"
Spike. Problems that I don't know how to solve I spike first (I write code with no tests, or with tests only as drivers of execution). That's easy. The hard question is, "Now I have all this code, I have to throw it away and TDD it?" That's pretty hard to stick to in a business environment. I choose to write tests-after for all those pieces of code that already meet SOLID metrics, and rewrite the code (using TDD) for the pieces that don't. The pieces that don't are very difficult to write tests for after, and they also happen to be the pieces where I find bugs (for example, I'll cut and paste a bit of logic and find its wrong for one set of inputs).
"All code should have full coverage of automated tests"
All three agreed that this is the case. Fowler: "If you have a full suite of tests I don't care how you got it [TDD or not]". I don't know about you but I'm still fighting this battle. I also have to deal with teams that have a "full suite of tests" and 80% test coverage, but where every single one of those tests simply executes code. No actual "test" occurs. Indeed, in particularly memorable test, I managed to delete 70% of the lines of code and all the tests passed (including deleting the one line that was the main purpose of the method). Approximately 90% of all the tests were complete garbage: they reported success as long as the code didn't throw an exception.
There are many generally good principles of design, the key is "generally", because when taking to extremes, they eventually conflict with each other.
Same applies to TDD, you have to know what design decisions it lead to are good, what design decisions it lead to are bad.
We also have another colleague, he wants to have his cake and eat it. His name is David. He is now doing septagon testing. Does it benefit the project? Probably not, but it gets attention, and that's the important thing. For him.
Then finally, we have a colleague, who as always remains nameless and faceless. We never rwally notice these colleagues. They produce working code, with a few sensible tests as appropriate. They deliver for the client. They are not arching for book sales, conferences, blog spam clicks or anything else. They are the real professionals.
Cyclomatic Complexity is not a principle, it's a specific measurement of potential complexity. Law of Demeter is a guideline SOLID is an object oriented-centric set of principles.
These are 3 completely different things.
You then go on to state using these makes software more testable. Since TDD can only exist in "testable" code, TDD is good design because these measurements/guidelines/principles are good design.
It's a non-sequitur of epic proportions.
I guaran-goddamned-tee you I can write a piece of software with less cyclomatic complexity than the Linux kernel, that is not nearly as well designed. Conflating those two things is what it means to be PHB.
CC is simply a measurement of a specific type of complexity, and like all measurements, it means jack shit without context. For example. the measurement of 8 inches. Is that number high, low, or normal? Or put another way, are we measuring a mans penis, a mans leg, or a mans hand?
this is why so much software turns to shit. This right here. The process by which you make decisions. If it's a good process, you'll have a tendency to make good decisions. If not, as illustrated by the post above, you will have a tendency to make bad decisions. A series of mostly good decisions will result in acceptable to good software. And the opposite results in unacceptable to bad software.
You want to learn how to write good, stable software? Learn to examine your thought process, and be explicit in your attempt to make good decisions. Be willing, and able, to identify bad decisions, why they're bad, and what you should have done instead.
What you did was believe the conclusion (TDD is good design) and then constructed the argument for it. DO NOT DO THIS. This is the stuff of bad decisions and software design failures, and this will happen regardless of which process you subscribe to.
The only part of this post that I think really deserves a good response is
You then go on to state using these makes software more testable. Since TDD can only exist in "testable" code, TDD is good design because these measurements/guidelines/principles are good design. It's a non-sequitur of epic proportions.
That's not really what he said. He was saying that essentially there is a sort of design isomorphism - like a mathematical dual - between code test-ability and other desirable properties like compose-ability, low coupling, re-usability, and maintainability. The argument in favor of TDD says that building code out of a TDD process that requires testing means making design decisions that have this nice property of building highly compose-able code.
> That's not really what he said.
It is what he said, here's his wording, verbatim
_What I've noticed is that TDD results in code that scores well against these measures, while code that scores well is easy to test (i.e. after writing the code). Therefore, I think the argument that TDD creates bad architecture is false._
You're interpreting things that were not said.
The entire argument is a non-sequitur. It doesn't even matter if you agree with him, an honest evaluation of what he stated would result in the conclusion that the conclusion absolutely does not follow from the argument.
The only thing that can consistently produce good design is good decision making. CC doesn't tell you anything without context, SOLID can be misapplied and is very OO-centric, and the Law of Demeter isn't even a principle, it's a guideline. It's like saying 'prefer composition over inheritance unless it's a clear win'. Ok great, but that doesn't actually result in a good design, it's cautionary guidance on what tends to be the better decision.
none of these things necessarily result in good design, and none of these things are required for good design. Hell, even the idea of 'good design' is nebulous and changes from 1 project to the next, and over time w/i the same project. Good design in a mobile app where energy is of the utmost importance is not the same as good design in a scientific application where correctness and verifiability are of the utmost importance.
The problem here is that you have yet another person coming to a conclusion and then working backwards in order to justify the conclusion. This is the sort of thing that consistently results in bad design __regardless of how many acronyms you follow_.
There is absolutely nothing in those ideas that intrinsically results in good design, or even intrinsically avoids bad design. That too was a part of DHH's point, one that a lot of people seemed to miss.
The conclusion does not follow because the conclusion came first. You mischaracterizing me as angry doesn't change that, but it is another indication of a flawed thought process (that the validity of the argument somehow stems from me being angry or not). Which brings us full circle back to the sort of thought process required for good decisions.
mreiland: angry rant that seems only tangentially related to jamieb's comment
me: I'm going to say jamieb has a sensible opinion on the subject, though I don't entirely agree with him. And I'm going to assume mreiland has had a bad day, but he hasn't said anything that contributes anything to my understanding of the subject, jamieb's comment or discussion in general.
The point is similar to DHH's point, in that a process (in this case TDD) doesn't magically give you good design. Neither does following SOLID, understanding the Law of Demeter, or being aware of the Cyclomatic Complexity of your project.
jamieb thinks TDD results in great design (the conclusion). jamieb then decides to cast around finding a reason why. jamieb comes across a series of completely unrelated things that "makes things more testable", and since they're generally associated with positiveness, concludes that TDD isn't negative.
The conclusion does not follow the rationale. The point is that this is indicative of the sort of thought process that tends to create bad designs.
Rather than trying to push for more developers to be pro/anti-TDD, we need more developers who are simply aware of the decisions they make, and the process they use to come to those decisions. That will have a far larger effect than any amount of TDD ever will.
Is it hard ? Sure.
But in my personal experience, most programmers err heavily on the side of failing to test thoroughly something that will become a long-term solution, not on the side of over-testing their throwaways.
You can write a good set of tests when you know the initial requirements, and not all requirements iterate at the same rate as the code. As you learn new requirements, you write new tests. The old ones don't go away (nor do the early requirements if they're good), even if you completely replace the implementing code.
Throwaway prototypes are part of the requirement discovery process.
In (most)JS and/or python I would just monkey patch the save method and/or constructor on the employee object to get the isolation and branch coverage.
I would probably need to do some of the same crazy abstraction for Java though.
public Employee create() { ... mapper.save(employee); return employee; }
Done.
All you do now is create a mock employee mapper in your tests, and you can verify just what you want in the Controller instead of anything else
You've got * Single responsibility classes (Controller manages employee creation business logic; Mapper is a dumb database wrapper; If you want to go even further, use a Repository instead of a Mapper to introduce one more layer of abstraction between the data model, that way you can slide in caching or in-memory replacements, or what have you at will) * Clean dependencies. There's nothing hidden about what this class needs to get it's work done
Guice is a representative programmatic example. Reasonable idea but once you trickle it into your code, it becomes a cancer quite quickly.
I believe it comes down to the team. Who is on it, how the team works together. If there are a lot of people on the team who enjoy working that way, they'll use it. If not, they don't. It lives on.
Answer: obviously, it is not the best thing on this planet.
Like "use the right tool to solve a problem", use TDD if it solves a problem.
That's a pretty easy logical fallacy. For instance, I've encountered developers who don't use version control. That doesn't imply that you shouldn't use version control because everyone doesn't.
To do so on an IDE with local history, you'd have to Ctrl-Z backwards all changes (not meaningful commits, but even small changes) linearly.
Don't take IDEA local history as a replacement to version control, rather a supplement.
His argument wasn't usage, it was controversy. Presumably, these developers you've encountered aren't announcing that version control sucks and you shouldn't use it. They're just not using it themselves, and you'd probably be right if you guessed that they wouldn't have a rational reason for avoiding it. Contrast this with TDD.
I've seen people argue that. The fact that people argue something is bad doesn't mean that it is, in fact, bad.
Only on average. We are fully capable of being irrational as large groups...
That's exactly the ad populum fallacy, so while you could do it, you probably shouldn't.
Or some people are stupid. The truth doesn't always win out.
> Like "use the right tool to solve a problem", use TDD if it solves a problem.
You could say that of anything; it doesn't actually help decisionmaking.
How about dialing back the religiosity and ask "is TDD useful?"
It is easy to take for granted that when you have a function like this:
function foo(int x, string y) { int count = x * 100; print y + " " + count.to_string(); }
...that if you send in a string for x or an int for y the function will blow up. And if you are using a compiled language, it will yell at you before you even attempt to run the code.
In ruby the equivalent code would run just fine, even with generally bad input and it is totally up to you (the programmer) to police yourself to write code that doesn't send across bad data.
A ruby programmer (and probably the same goes for PHP and Python), will end up writing a fair amount of tests that simply ensure that the boundaries are enforced properly and sanely in the code.
There is a lot of love about ruby, but if you are going to write safe, dependable, bug free code it is up to the programmer to enforce boundaries, do validations, and so on and so forth and then write tests that validate that they are still correct over time.
Part of the reason this argument isn't so much happening in other languages is in part that there is maybe not as strong of a culture of TDD in other languages, but also simply because some of the problems don't exist in Java, Scala, C#, etc. because of a type checking compiler and stronger boundaries between components.
A lot of the things that we consider "good architecture" are simply the way we name things, the way we put files in folders, and where we place boundaries in our code. We could write everything as a single file that ran sequentially and jumped to different locations (which is largely what happens anyway), but to make things reasonable and understandable we use patterns like MVC, or composition based functional programming, or MVVM, or various OOP patterns, and so on to express program logic and create logical, sensible boundaries in our code.
If you see the world through the lens of naming things, file structure, and conceptual boundaries, you will see this argument about TDD is more about how strong the boundaries in our code should be and where they should live, not whether TDD as a tool is a good thing or a bad thing.
Rails MVC sees the world as 3 tightly coupled things - model, view, controller. And in that tightly coupled world(along with weak ruby boundaries), TDD is painful and probably not worth your time. DHH is right about that 100%.
If you don't structure your project as Rails MVC does, TDD can be a very pleasant way to build your project and have confidence as you change it over time. Ruby is not the best tool for that job, but it can be made to work. In general, the weaknesses of ruby's boundaries will mean this argument will keep coming up over time and will never be solved.
Small groups trying to deliver on aggressive schedules waste too much time with test-first.
Having a competent leader and skilled team that can follow SOLID principles removes the need to be test-first.
TDD is not dead. Test-first probably is.
http://stackoverflow.com/questions/3192090/test-driven-devel...
It also didn't help how TDD was shown in a study to not work out very well. I don't have it off hand, but it was talked about in the "Making Software" book http://www.amazon.com/exec/obidos/ASIN/0596808321/wwwicefoxn...
If you write the code first, then try to write the tests after, you may have to refactor code to support testing (if you're not used to writing testable code), or you might end up having to do very convoluted tests to make them work (see above re: 3-level mocks), or you might just accept subpar test coverage
Yeah, it takes a long time to become competent.