TDD Derangement Syndrome
blog.objectmentor.com
blog.objectmentor.com
Bottom-up programming develops tiny functions that are each trivially correct. These tiny functions form a base on which the next layer of tiny functions is built, and so on -- each layer written in the layer below. Most of these tiny functions begin life as tests, which are then generalized and encapsulated in a function. So in that sense BUP is sort of test-driven. But if it's necessary to write further tests to determine the correctness of these tiny functions, You're Doing It Worng.
In exploratory programming you're not sure where you're going yet. The development path during exploratory programming looks like a crooked tree with lots of stubby dead-end branches. If you first developed tests for each of those stubs, only to find that that's not where you're going, you may never stop writing tests, assuming you even know what to test for. Plus, testing during exploration would kill the flow. You can't be a pioneer and a settler at the same time.
I'm not saying not to test. And I'm not saying not to use TDD. TDD is probably great for top-down projects where you already know where you're going. But if it preempts the process by which many of the great programs were written -- if it preempts hacking -- then it's not the final word.
As for the exploratory programming argument, a key element in TDD is to write one test and the make it succeed. You are not supposed to write a lot of tests and then the code that makes them succeed. So unless you are so exploratory that you don't even know what you want your next function to do, this would not hinder you either.
That's like saying type systems are good because they help you write sound proofs about type systems. I know they can be really useful, but it's an incredibly circular argument.
I'm sure it's a highly useful tool in some cases, but the people pushing it uncritically make it harder to tell when it's actually a good fit. In my experience, it's very useful for safely making changes in a legacy code, but I suspect it can encourage over-engineering surface details or getting caught in local maxima when used for initial design. (I usually just design with a REPL or notepad/whiteboard.)
The "red green refactor" loop is more like:
1) What's the next problem this software needs to solve? 2) Write a failing test that demonstrates the missing need. 3) Implement said behavior. 4) Refactor until you don't see anything else to refactor.
I think step #1 is one of the most important and yet I don't see much discussion of it. Pick the most "key" tests to write first, whether that be your happy path or the functionality that has the most risk.
On refactoring... I work with people who TDD all of their work and other who write unit tests afterwards. In my experience, those who test-drive are more comfortable refactoring otherwise. When I was test-driving imvu.task ( http://thespeedbump.livejournal.com/63798.html ), I spent a great deal of time just staring at the code after it was written... it's very continuation-heavy code... and suddenly I saw a refactoring that collapsed it by 1/3. We've hardly touched the code since. It solves the right problem with the right amount of code and I think it's only had a couple very obscure race conditions.
In summary, TDD is one of my favorite tools for focusing my efforts so we build what we need, but only what we need.
I know the Rubinious project was built aiming to use TDD and it doesn't seem to have gone anywhere YET. As far as I can tell, a compiler is one beast that needs a strong abstract design first rather than a set of tests first - but I'm prepared to be surprised.
Pivotallabs lists Twitter first but I would be rather surprised if someone could verify that a substantial part of Twitter's app has been built using TDD (go ahead, surprise me).
But since you're looking for open source success stories there was some very positive noises about TDD from a Twisted dev recently:
"I would strongly recommend at least practicing some TDD, too. Learning how to write tests was useful for me, but practicing TDD really expanded my design skills in a bunch of surprising directions that were almost unrelated to testing. I originally felt as you do, that I wasn't disciplined enough to do it; now I feel like I'm too undisciplined not to do it ;-)."
From: http://glyph.twistedmatrix.com/2009/09/diesel-case-study-in-...
Though he specifically states that Twisted didn't start out TDD, which he regrets, but maybe that disqualifies it.
Regarding Rubinius (and JRuby that was listed in the original post) I'm not sure trying to create a spec for an existing language implementation and then try to meet it would count as "true" TDD if people care about such differences. But maybe they do TDD for other bits lower in the stack, I'm not sure.
I've only scanned the Rubinius code and noted their blogs/announcements but ... it seems like they shifted from one basic model of their virtual machine to another in the middle of their process and I'm quite skeptical that this either indicated they doing well or that it would be considered a victory for TDD. Looking from the outside, it looked more like "we thought that TDD would let us get by without design but we were wrong...".
I've heard TDD described as "write a test, write a function, repeat" and THAT is not enough. You definitely need a guiding design as well.
I've never used or looked much into TDD, but it seems to me that that's where TDD would be best: areas where input is easily done programmatically & where output can be easily checked for whatever property. How does one test a big GUI app - write some sort of binding to xlib to move the mouse around and go through routines? I'm sure it's possible, but I suspect it's pretty difficult to come up with useful but reliable tests.
There is a big lack of TDD in highly succesful Open Source endeavors. Part of this is perhaps because a failed project never get attention. But also note that defects in Open Source projects are good. They instill new programmers to come to the project and work on it because there are some easy low-hanging fruit. Furthermore, as exemplified by Keith Packard recently (see http://lwn.net/Articles/354408/ -- which unfortunately is behind the Iron Curtain for 24 hours more). Keith mentions that he sometimes deliberately breaks some old code by introducing obvious bugs into it. Then, when nobody have complained for a couple of years, it can be removed on the grounds that it was not interesting.
With rigorous testing this is unachievable.
My note about type systems doesn't have anything to do with whether it's correct or not - just, if the first example people give for the usefulness of type systems is more type systems, it makes me wonder how much consideration the speaker gives practicality. (Then again, I'm not a grad student...or an agile methodology coach.)
(sarcasm)
What I don't get, if Robert Martin uses TDD, why doesn't he simply list those "real" projects (real as in projects for clients) , he's worked on?
Edit: (e) Listing projects that are: 1. real (not just student, testing or open source projects); 2. large enough that real stakes are on the line (I want to see an example of a 2 MLOC system before and after TDD, not the 70 KLOC of Fitnesse) would make me pay attention. Then I want to see how TDD improved the state of the project (and that picture has to include things like productivity and cost analysis of test first testing everything). Then I want to see why the same results could not have been achieved with "test last" or "test whenever" unit testing. Then I would want to see an explanation for the rate of success other people have had without ever practising TDD - because if TDD is as effective as the zealots claim, and if the zealots make a living making those claims, then that explanation HAS to be there.
"An Initial Investigation of Test Driven Development in Industry" by Boby George and Laurie Williams, and "Realizing quality improvement through test driven development: results and experiences of four industrial teams" by Naggapan et al.
and I am not too impressed. The first study has 6 TDD teams and 6 control teams. Our null hypothesis is that these teams produce exactly the same quality of code, measured in defect rate. Now, any sane study would take a look into (statistical) power, ie the probability the test will reject the null hypothesis. If we further look at the box-plots, we see that the overlap between the "treatment" and control group samples is pretty big. Hence, we will need to test for any statistic significance before coming to the conclusion. In fact, it may even be that we can reject the idea that writing in a TDD style takes 16% more time on average.
The second study is much worse. It quotes studies secondhand, only has 4 studies (with 4 "controls" being completely different systems). It generally fails to the problem of lurking variables, or confounding: That some unmeasured external factor affects the study. For instance, it is put forth that the Microsoft teams in the study employ many methods for minimizing program defects including static analysis. The other grande problem here is that there are no assesment of the control-group systems at all. They will, invariably differ in more than one variable and the question is: Have we measured to benefits of TDD or have we measured the benefits of something else? We can't say for sure.
Finally, we only test statically typed languages with a subtyping construct (Java, C#, C++). I think (and I stress it is a hypothesis) dynamically typed languages would benefit considerably more with a TDD system in place, and I hypothesize more expressively typed languages (ML, Haskell) would benefit less. But I have no good idea of how to design an experiment that can show this hypothesis true.
It makes me even more skeptical of TDD.
I would expect most projects using just about any new methodology to look notably better in tests through the programmer's equivalent of the placebo effect: a new methodology imparts energy and optimism compared to just doing the same old.
This would lead me to deduce that once the "shine" wears off TDD, it would actually give no improvement in quality and result in considerably longer development time...
Person A claims "Process P is good!"
Person B retorts "Sez you, can't prove it to my satisfaction!"
And the debate rolls on and on about whether we can prove P or not. I am not blaming Person B, but the question I want to put to them is this: Can you prove that whatever you are doing now works? Or is it just a case of you feel it works for you?
Good Question.
Another question is "Do you go around calling legendary programmers who don't use your practices, yet ship product, "Stone Age Programmers"? Or the questioning of your favorite practice X as "X derangement syndrome"? If so why?
But if it amuses you, consider the question of whether people like Alan Turning and John McCarthy operated in the stone age of computer programming.
For that matter, consider the question of whether we are still in the stone age. I am inclined to say we are remarkably unenlightened. Some people ship without process, we don't really know why other than to make vague statement like "they're legends." Others have process but fail to ship and we again don't fully understand why.
Although our tools are concrete, our application of them seems to be a mystery with very little hard research or true science behind it.
Oh I never said or implied you did. That is why I phrased my comment as "Another question .
I unreservedly apologise if anything i wrote here I gave that impression.
", consider the question of whether people like Alan Turning and John McCarthy operated in the stone age of computer programming."
but those are not the people being addressed as "stone Age programmers". The people interviewed in CaW (which is how this hole brouhaha started in the first place) and Joel Spolsky are hardly "Stone Age" programmers just because they don't "evolve design" using TDD.
As per Bob Martin , people who don't use TDD are "Stone age". People who question the value of TDD are "deranged". This sounds like religious fanaticism to me.
To extend your original comment.
"Person A claims "Process P is good!"
Person B retorts "Sez you, can't prove it to my satisfaction!""
Person A replies "That is because you are a deranged Stone Age Programmer".
My question is about people who reject some things for lack of empirical justification while accepting others without such justification. If you want to piggy-back off my question to criticize famous consultants, how about asking whether Robert Martin is actually Person B and not Person A?
I haven't read his work in enough detail to comment one way or another, but I certainly am open to the possibility that some of the proponents of agile methodology do so on the basis of very scant evidence while rejecting other approaches for lack of substantial justification.
I never said it did (I am saying this the second time). You have a right to your opinion.
You were using a rhetorical device to put forward an opinion. So was I.
I wasn't aware it was somehow offensive, especially since I made it clear I was borrowing the device and not attacking your points and apologised for any inadvertent impressions to the contrary. . If that still offends you ... [shrug] whatever.
"I haven't read his work in enough detail"
I have.
"but I certainly am open to the possibility that some of the proponents of agile methodology do so on the basis of very scant evidence while rejecting other approaches for lack of substantial justification."
Fair enough. With that, I exit this thread in the interests of harmony and not wasting time hairsplitting.
For example, if you think unit testing is generally good but TDD is a bad idea then give some context to that: is TDD worse than no unit tests at all? Is it worse than low unit test coverage say 20, 30, 40%? Or off at a tangent TDD makes Java programming feel like Ruby (or vice versa), just something for people to intellectually grab onto in terms of scale.
I ask because not unit testing at all seems (to me) to be further from unit testing than TDD is, yet from this current discussion it would appear that unit test fans have more issues with the latter than the former.
There's that ever-elusive "it improves design" argument for TDD, but that one's far harder to prove either way, and I've personally seen it cut both ways: sometimes things end up more nicely decomposed, and sometimes they end up wayyyyy too decomposed.
As for the design argument, I think that the thing that TDD actually encourages in terms of design is testability (something that I do think is desirable, in general). If you have high test coverage without using TDD, you've probably either taken testability into consideration before you wrote your code or you made some changes after the fact to accommodate your tests.
The places I've seen TDD lead to less-than-desirable design is when some larger problem is broken into such tiny pieces with so many interfaces that it becomes hard to read through the code and follow the flow of control. That's certainly testable, but often times I personally get as much mileage out of writing tests one level up, against some higher-level abstraction or API that those other things are all components of, and not being so religious about breaking up all the little parts. In my experience those slightly-higher-level tests (they're not even "integration" tests, since they're still strictly against one logical component) tend to provide the same amount of value in terms of preventing regressions while requiring fewer compromises to make the little parts testable and giving more flexibility as to future refactoring. Essentially, you're treating a whole bunch of classes/parts as internal implementation details.
This is why I prefer the rebranded "BDD" as opposed to the original "TDD".
I find that the perspective that BDD offers makes me write much more useful tests, since the ultimate reason we write objects in the first place is to use them, right? I don't care about how it works inside, I care about how it gets used. There's nothing technically different between BDD and TDD, it's a matter of philosophy, and I find BDD's take on things to be better.