What TDD is good for
theaxolot.wordpress.com
theaxolot.wordpress.com
This is my perception as a non-TDD person. Maybe someone can validate if this makes sense or not.
But I have colleagues (working in other projects) who told me how they are succesfully applying TDD to refactor an old codebase. They start with big, not-so-"Unit" tests to make sure that they don't break the thing at any step, and then they move to more specialized tests when tackling specific parts of the code.
Nothing in your example prohibits or even hinders TDD. The only thing required by TDD is that you specify an interface and add tests to specify what behavior you expect from them. The moment you have something you can compile and run, your TDD path is completely open.
Also, I'm noting the hints of the same specious reasoning that plagues discussions on microservices, mainly the "it's-all-or-nothing" fallacy. You can benefit from TDD if you, say, are writing a RESTful interface and start off by only adding unit tests checking for controller behavior. If your project can live with 0% line coverage, it can certainly live with TDD buying you 20% line coverage.
The codebase is mature and it's architecture is likely well established. It's easier to write tests for small changes in a very established system than it is to write tests for a system that's still in flux.
In most cases that I've seen, this isn't done because there's no financial upside to doing so. When your bonus is tied to velocity, not quality, any effort on testing is too much effort.
TDD stands for "Test-Driven Development" not Design. Your design could change radically after creating the initial tests, they should not be used as a way to freeze your system unless there really is a true finished and complete state for it. (This is true for any test suite, not just those developed doing TDD.)
Which implies a cost of at least 2x in dev hours. One should always evaluate if it is worth double or triple LoCs to maintain.
If you don't bother with tests, discussing TDD is like famine victims discussing nutritional levels reported in food labels.
> My tests are usually bigger than my implementation.
That's perfectly fine if you feel your tests are important. Getting back to the topic, TDD does not enforce coverage targets or even minimum coverage bars.
No, not really. I have no idea what led you to believe that.
Think about this for a moment: does TDD force you to comply with a target coverage metric? Is the TDD police going to knock on your door if you leave out a usecases? What about a whole component?
How many seconds does it cost you to add a test checking for the happy path?
Of course it does. You only write production code to make tests pass. Hence 100% coverage.
When would any not-covered code be written in TDD?
No, not really.
TDD just requires red-green tests when adding features. You can add a test that only checks for the happy path.
More to the point, nothing forces TDD practitioners to succumb to the hypothetical slippery slope fallacy of having to follow a fundamentalist path 100% of the time. Why do you believe that nonsense if even unit test best practices stay clear of 100% test coverages?
Yes, really.
If you know what is TDD then you wouldn't make mistakes such as believing each and every single unit of code must have a test. That's specious reasoning and slippery slope sloppy thinking.
> But then any discussions about the properties of TDD are pointless.
Not really. You're just succumbing to slippery slope sloppy thinking. Adding a red test and working your way up to a green test is self explanatory. Being contrarian just wastes everyone's time.
TDD is test driven development but it should not freeze your system. If you make system testable it is the opposite of frozen. So if you design system to be testable it already should be more flexible.
The design here is one notch of abstraction above.
Also tests should not freeze anything as they should be removed if not needed. If you write 20 tests during development it doesn’t mean those 20 tests are needed for future maintenance.
Tests don’t help you design software, they help verify properties of the implementation. For non-trivial greenfield systems, the full set of properties (which is a very high dimensionality tradeoff space) and their expression in the implementation isn’t really knowable ahead of time.
Also, this is not specific to software. Defining how you will validate a system, before you go build the system, is simply good engineering.
Hardcoding specific implementation detail assumptions into the design of complex non-trivial systems so that you can write a test is a brittle way to go about writing software. There is a reason that waterfall-style never seems to produce good systems in a reasonable amount of time or money.
I write non-trivial greenfield systems and put inordinate amounts of design work into them (literally years in some cases). I have never found a case where that allowed me to write useful tests prior to implementation. At a minimum you need a complete architectural skeleton that runs and has had its non-functional properties verified analytically before you can write tests that you’ll likely use.
I’m not against writing tests. I invest far more in testing than most devs due to atypically high reliability and robustness requirements. I’m against the complete waste of time that would be writing tests before there is sufficient implementation to design maximally efficient and appropriate tests. Testing of some software is extremely expensive in both time and money, so optimizing test effectiveness and efficiency becomes valuable.
Starting with TDD is like being a general whose recommendation to every international politics issue is to commence training for a combined arms assault on the capital of the country's historical enemy, because that's something that can be iterated to provide good estimates and error-free execution. Except, that general misses other, possibly much more effective approaches, such as airstrikes, assassinations, supplying domestic terrorists in enemy territory, or... not shooting anything at all because we're not at war and the problem can be trivially solved by respective VPs over some drinks. Or the problem isn't even a problem.
Keep the TDD General around for when you're actually fighting a land war - but for the love of $deity, don't let him drive international policy in peace time.
Or, it should be anyway. People talk about how it "drives" design but I dont think it should. I think tests should be as design-agnostic as possible, providing freedom to redesign.
I always start with an end to end test. These dont force you to commit to anything.
I don't agree. TDD is only relevant regarding the interfaces used by units of behavior. Software architecture isn't something that you design or verify or even assess with unit tests. Software architecture is purely a design task, and redoing software architecture is a kin to rewriting the whole project.
In performance-engineered systems, a component’s behavior in isolation is not its behavior in the system, and that behavior determines correctness. Basically the same fallacy as most micro-benchmarking. Resource availability and locality vary dynamically, and the software design has to account for this, which is extremely non-trivial to do without code you can measure in the system context.
And yes, modifying software architectures to address design issues that show up in implementation details or to slice some non-obvious tradeoffs differently is expensive, but is almost always required to some extent. That’s a cost of business if you are doing non-trivial greenfield software, it comes with the territory.
If anything it helps.
For example, systems software is mostly about testing system behavior under various loads and resource constraints. The ability to even measure that reliably without the measurement interfering with the software operation is dependent on a lot of implementation detail.
Call it what you like. Some people would be insistent that this is TDD. Some would say it isnt because it's not test driven design. Some say it is TFD. I dont really care what it's called, for me it doesnt make sense to develop serious non-spike/non-POC code any other way.
It's one of the few things in tech where my opinion has hardened and not softened over the years.
"Unit tests" on the other hand, blech, sometimes useful, very overused.
Yes, my thoughts exactly. At its core, TDD forces a workflow where the interface is the first thing laid out, and the expected behavior follows. Therefore when designing new components, it becomes readily apparent when the interface doesn't work, even before anyone invests any time adding the missing behavior.
The main idea of writing a test which fails and then fixing it with the code change applies almost anywhere.
E.g. when fixing a bug, write a test case that fails because of the bug, and passes due to the bugfix.
You don't necessarily need the refactor step, though. The continuous refactoring is the greenfield aspect, perhaps.
Refactoring can leave a trail of TDD tests that used to be independent, but are now tied together and redundant.
i would also add regressions as a place where TDD is excellent: reproduce the regression with a failing test, then fix it
it is exasperating that TDD advocates will often not admit that there are any tradeoffs or contexts in which it is more or less appropriate, and I think it does a lot of damage to the reputation of the technique (which can be effective in some contexts) as consultants snake-oil
Not just regressions, but bugs in general.
If you can reproduce your bug in a test, then verify that your fix causes your test to pass, that's a great way to build confidence before trying out your code in production.
And in fact, this is mentioned in the article as part of maintenance:
> Writing a test first to prove out a bug before you fix it is pretty neat.
This seems to assume that rewriting your tests is slower than rewriting your implementation which has no tests. But if you keep things bite sized, and the logging you collect from tests is good enough, it's often faster to change a test or a type and have an LLM rewrite the relevant part of the implementation.
Also, I think we have this bias where we get so excited about the implementation that's blossoming in our mind that we forget to check really important stuff before writing that implementation. A bias towards rationalism and away from empiricism, or perhaps towards our own aesthetic leanings and away from those of our peers.
I like writing a few tests first (even before the types or frameworks they're made of even exist) because it puts me in the caller's mindset. With those few tests to guide me, I can then let myself have a lot of fun building something with the callee's mindset, and tests will hold me back when I've gotten too creative for the caller's needs. If I try to just build it without those rails, I end up with something really interesting and not very useful.
A lot of the maintenance/brownfield benefits the author cites can also come from the compiler itself if your language has branded types, ADTs, non-empty collections, and whatever else. I feel like you shouldn't bother testing what you can already prove.
None of this is esoteric. It can be spun up in TS, but mass culture is lagging behind technology.
If you write a square root function, static typing can just ensure that it always returns a number which doesn’t tell you much. Unit tests can ensure it returns the correct number at least for some inputs. This is a lot more useful for ensuring correctness.
The counter argument is that tests donsnt ensure the function always returns a number. Maybe it returns a date or a string for some particular input which is not covered by the test? While this is true, in my experience this is not a common problem.
webdev isn't square root functions it's working with complex data APIs.
In fact, the argument was about ruby/rails not needing types/interfaces was because "the tests are the interface" that's how baked in tdd was
Oftentimes, type is just a less specific match. It would be a waste of time to write a separate test case that add(2, 2) produces an integer, rather than just testing that it produces 4.
I've hardly ever seen Lisp test cases that look only at the type of an output, rather than a specific value. Even a function whose job is to operate on types actually has inputs and outputs that are values representing types; e.g. the symbol string representing string types or whatever.
However, there are tests which show that a function signals an error when given wrong types. Those would not be necessary in a static language; the function called with wrong types would not compile, and it goes without saying that all tests must compile, unless we are testing the compiler.
As someone who has programmed in statically-typed languages and ones that aren't, I can absolutely, positively confirm that a static typing system and compile-time checks absolutely, positively replace an enormous swath of tests.
The difference between refactoring code in Ruby and in Kotlin (or Java or C# or whatever other language you choose) is worlds apart when you can actually trust that the thing being sent to the method you're checking actually conforms to a certain type.
What would take multiple days and maybe weeks to find all the edge-case-hells of type assumptions in Ruby is often a multiple-minutes change in those other languages.
Unfortunately, this industry (especially startups) embraced the low-end of several untyped programming languages that are used in production-grade systems and evidently there's a price being paid by those choices.
Java, Kotlin or even C# would be a much saner list of statically typed languages to use and testing code written in those languages is much easier to reason with.
But my argument is you would not write such a test in the first place. You would write a test which checks the output is correct which obviously implies it is also a number. Static typing does not in itself check if the output is correct, so you still need the test.
And, there are huge qualities of life and developer velocity benefits to the statically typed languages due to those.
I’m just pointing out that static typing does not replace any tests. At least not any test which a reasonable developer would write. What competent developer would write a test that only verifies that sqrt outputs a number, but not compare the output to an expected value? But such a strawman test is the only kind of test static typing would make obsolete.
Of course you need to test for NaN, Infinity etc because these are number-like but still invalid. But you also need to do that in a statically typed language.
I don't even have an axe to grind about TDD itself. I don't do it and nobody forces me to. I'm just hypothesizing that the practice was tied to the primitive typing of its time. I could be wrong about this.
What does your square root function return if you feed it a string? And what does it return if you feed it a JSON object?
If it is JavaScript you probably get NaN.
If you have code which pass a json objects to a sqrt function then the calling code is in error so presumably tests for that code would flag it.
Counter question: What happens in a statically typed language if you pass a negative number?
I just tried in C# and surprise: NaN
The value added by statically typed languages is that your runtime errors are instead compile time errors what are checked for free without having to add a single test.
You do, but not the boilerplate stuff to check that your code throws a runtime error if you pass a JSON doc to a square root function.
Having runtime errors instead of compile-time ones is also a major liability.
> A sqrt function returning 27 for every input would be “proven correct” by static typing.
That's not what's being argued.
Have you often written this type of test? What value do you feel it have given you??
The whole point of tests is to ensure your code doesn't trigger runtime errors caused by contract breaches. You are not describing a solution, but the actual problem you mush avoid. It's baffling how you don't realize that.
> Have you often written this type of test?
If you believe this is a foreign concept to you, I don't know what else there is to say.
> What value do you feel it have given you??
Are you actually asking what's the value of not having your code throw exceptions because you write broken code such as passing JSON objects to square root functions?
Of course a caller passing a json object to a sqrt function would be in error. This error should be discovered by testing the component which does that.
> The whole point of tests is to ensure your code doesn't trigger runtime errors caused by contract breaches.
IMHO the purpose of tests is to verify that code works correctly. Of course this implies it doesn’t throw unwanted exceptions but that is only a small part of correctness. Sqrt in not correct just because it doesnt throw an exception on a numeric input, it have to return the correct value.
Testing that a component or function works correctly also automatically implies that alle necessary internal contracts are followed. The reverse is not the case.
bazoom42 touches on the value testing aspect that goes beyond the mere type or return.
Typing and compilers also only guarantee coherency inside your declarations. If for instance you're expecting some library to return an int, but it's utterly broken and actually tries to return a string, the strong typing system will only let you identify why it's crashing in production. Hopefully you'd want to detect this class of issues beforehand.
If I try to call a third party library method in a language with a sane type system (like C# or Rust) that returns a string and attempt to assign that value to an int, it won't even compile. And I'll get a red squiggly line before I even try to compile.
What you're saying would only apply in something like Python or TypeScript where you might be using types/type annotations but the library you're calling isn't.
You're assuming that for instance the library is correctly setup and won't crapout because a system requirement is not met. Or that no other external factor will interfere with the result (e.g. your library is hooked to an API)
Compile time checks will be against a declared interfaces, not the actual library built outside your project.
To me that's the kind of issues where having a test coverage really helps, a lot more than just checking that mocks and interfaces match each other (to show my hand, I prefer dynamic languages, so interfaces will rarely fail and many edge cases can be simply be coerced into a generic path, so the real scary stuff is when the rubber meets the road and expectations meet reality)
Correct, and if that isn't the case then that's a problem with the library, not my code. In C# I've had none of the issues you just mentioned with libraries, with the exception of ones that expect some specific native DLL to be present that the C# library calls into. And unit testing won't save you from that issue because the system you're running your tests on isn't the same as the system prod is running on.
To be frankly honest I question how much you know/understand about languages with strong type systems based on your comments.
That was one of the first mistakes that began to spread everywhere and new engineers bought into the fads of using such unsound languages like JS, Ruby where it was difficult to reason with the type system to the point where they were testing the expected types themselves.
In some applied uses such as backend systems, the fundamental design of untyped / dynamically typed languages can just let down the quality of the software in the first place and be a massive burden and cost on the developers to the point where they MUST use TDD no matter what even in greenfield projects.
TypeScript has a more sound type-system, but still just serves as a temporary solution to the fundamental issues JavaScript always had and its still very easy to escape the type system and break it at runtime.
There's no replacement for educated engineers who pick the right tool for the job. If you go too heavy on static types, you get Haskell and Idris and get a lot of difficulty training people past the learning curve and actually putting value-generating code into production. If you go too heavy on automated tests, you get test suites that take hours and hours to run in CI and are extremely brittle and difficult to change. If you go too heavy on security infrastructure, you get big bank style practices where you're lucky to put anything past the review committees / bureaucracy.
Beck's book was 2003, with examples in Java and Python. David Astels wrote a book later that year, again primarily Java but also with short chapters discussing RubyUnit, NUnit, CppUnit.... Growing Object Oriented Software was 2009.
My guess is that "peak" is somewhere between 2009 and 2014 (the DHH rant); after that point there aren't a lot of new voices with new things to say ("clap louder" does not qualify as a new thing to say).
That said, if you're aware of the gap between decision and feedback, and managing that gap, I don't think it matters very much whether that feedback comes in the form of measurements of runtime behavior vs analysis of the source text itself. It might even make sense to use a mixed strategy (preferring the dynamic measurements when the design is unstable, but switching strategies in areas where changes are less frequent).
If you're touching the code in any way, there must be a way of writing a test which fails if you don't touch it in that way and then passes if you do.
It's not necessarily just about ensuring correctness of new changes.
What if you're wasting your time with that change? I.e. the test you want to pass after your change, maybe it passes already, so the change is useless. If you write it first, you will know.
The TDD test is itself compiled, and so the language features/checks help write the test. Then, if the proposed change doesn't compile, that's a fail? The compiler is helping again.
But if you want to draw your analogy to its absurd conclusion, I think a lot of people would hate the idea of needing to wash their hands to do a surgery remote controlled by a joystick, because the sterility of the instrument doing the operation is independent of the sterility of the hands remote controlling it from afar.
Software devs have not found out that anyone dies unless we do strict tdd. But we have found out that unless we have a good amount of automated tests, the software becomes brittle and hard to change.
TDD is more of a style that some swear by that has alleged properties making everything better, like the way doctors wash hands in a peculiar way. But just "washing the hands" thoroughly would probably also help people not dying.
On the other hand, there is no way to measure the "effectiveness" of TDD as compared to not using it.
I'm sorry that you feel that strongly about it but that certainly seems like zealotry to me.
There are definitely ways to measure, for example by it's impact on bug resolution time. The following study looks at static code quality, but it could be applied to TDD:
This study is about static code quality, but could be applied to TDD as well. https://www.researchgate.net/publication/359129462_Code_Red_...
> _for example_ by it's impact on bug resolution time
There are areas at the edges where the mocking/stubbing required to really follow TDD cause make changes harder but never find bugs. There are entire families of bugs that are far better handled via strong types than by building tests. In the right languages, there's functionality where some kinds of tests are just testing the library, but hard red-green-refactor mandates tests with negative value. We all have been in situations where a small code change requires 6 hours of changing tests for reasons that weren't tied to the real reason the test was there, but ancillary reasons. There are tradeoffs.
When someone asks me whether we should use TDD on a project, the right answer depends on what it is, which language it's written, and whether we are mandating it across the entire codebase, or there are specific things where we will ignore the worst cases. Are we writing payment software in Ruby? a data pipeline in Haskell? Making a bunch of API calls in Clojure? It's not all the same.
That's not TDD that's either a badly designed unit test or (more likely) you should have written an integration/e2e test instead.
>There are entire families of bugs that are far better handled via strong types than by building tests.
Tests and types are not solving the same problem and they work better in conjunction than alone. Types reduce the execution space, reducing what can go wrong and tests validate the execution space that does exist.
Yes, types reduce the need for tests by shrinking the execution space but this will be for scenarios which should never happen.
There are entire families of code and bugs that unit tests are unsuitable for but that's a whole different topic.
> but hard red-green-refactor mandates tests with negative value.
I have never seen an example of this. I've seen plenty of tests with negative value but they were simply bad tests - usually a unit test when an integration test was more appropriate.
I'm afraid that's not a TDD problem, but a problem created by nitpickers who love petty gatekeeping.
That goes out the window after refactoring though. Tests created before refactoring continue to pass, but they are connected to changes that no longer exist and therefore could not be backed out to show that the tests fail. (Not without rolling back the refactoring). Or, tests that were connected to independent changes now cannot be easily made to fail independently. Refactoring tied them together. Thus they are redundant.
It bring lots of benefits into large scale refactoring / maintenance in existing projects and totally agree with its less usefulness in most greenfield projects, with the exception of extremely safety critical systems that invest lots of money into the project.
Otherwise without the above, you would not attempt to consider using it on a new project with a very hard release deadline attached to it.
HOWEVER, IMO, having a set of behavioral tests (BDD) and a roadmap in the README helps a lot in staying on track. It's the best of both worlds: you don't get stuck in the quagmire of writing tests for things that don't exist yet, but you have just enough tests to keep yourself honest.
So they rigorously test all those logging calls?
logger.info(“Request:” + JSON.stringify(httpRequest));
Terrible! I removed it, deployed it, and…alarms went off. It turned out that the line in question was being used to populate a critical metrics feed.
In that terrible, terrible case, I agree that the load-bearing log line should be tested. In any other scenario I’d personally avoid it, outside of mission critical medical or other codebases. But that’s me, I understand the impulse to test it.
I’m providing this background because I’m still fairly wound up by a terrible interviewing experience I had with the only company so far that specifically mentioned TDD and made a big deal about it.
I’ve never experienced such a farcical technical test. It was an hour, and the interviewer (only one, red flag already as it allows for bias) spent twenty minutes debating the name of a single test and kept circling back to the topic when I tried moving past it.
Kept making a big deal about “TDD test names”. Refused to let me write anything. Refused to elaborate on any requirements or even what he really wanted me to do.
After the hour was up I’d written one working test and one broken test. The actual system under test was a one line method, because he spent another fifteen minutes getting angry that I wanted to even run the test.
Absolutely ridiculous and it’s unfortunately made me very wary of anywhere that makes a big deal about TDD despite being a big proponent of testing myself.
So yeah, if that’s people’s exposure to TDD I’m not surprised some people don’t like it.
If you’re ever interviewing for a UK based company that works in the arts and theatre booking space that mentions TDD, run a mile.
Spektrix
TLDR; TDD makes you a better programmer, even if you don't use it long term.
I have tried just writing end-to-end tests. I have tried writing unit tests after the fact. I have tried writing unit tests before the fact (TDD?). They all lead to problems.
I would like to lean towards a new model, that I would call "contractor-driven engineering" (CDD). The idea is to engineer software systems like a contractor builds a building. First you get a dumb person, who has the basic skills of a trade (so, your average software developer today). But then you bolster their naivete and inexperience with known quantities and an approved architectural design. The end result is a person with a basic level of skill, that can still be relied upon to create a reliable system.
This requires two things first, though:
1) The "known quantities" would be your well-defined, pre-produced, quantifiable engineering parts. Like how a 10d and 16d nail are built to a specification, so that when you architect a building frame, you know that the 10d nail will work in this one part, and the 16d nail will work in this other part, and you can be assured the building will not fall down under a strong wind. We do not have known quantities built to an engineering specification in the software world. We have software "things" that "an experienced person" says "should work", but if we built skyscrapers like that, everything'd be fucking falling over all the time (the way it falls over in the software world all the time, but nobody minds, until all of Delta's flights around the world get cancelled....).
2) The "approved architectural design" - we need people who are actually trained and certified as architects, to dictate the design to the software development team. I think in my 20-year career I have seen perhaps one team actually build software the right way (and by "the right way" I mean "not falling into obvious traps that anyone with enough experience has seen and would avoid"). That architectural training needs to involve the codification of a lifetime of engineering discipline (that is available, if you ask us old fucks for advice), combined with the "known quantities" above, so that designs actually meet stated goals and requirements. This, by the way, completely jives with the idea of "changing software all the time", because you can just ask the architect to look at the change and tell you if it'll work or not.
Why would I think this is better than TDD? Because TDD still assumes the developer is going to simply test their way out of bad decisions. You can't do that. But what you can do, is abstract away the decisions that are hard, and leave the work that is predictable and not hard.
I see a disconnection with reality here… Somehow bugs are caused by dumb people who thinks their code is great but it’s not. So we let smart people come up with practices that dumb people will follow and mistakes will be avoided.
How these certified architects folks will prevent defects is not clear to me.
We don't have that for software. But if we did, a lot fewer of our buildings would fall down. And it would actually be easier and faster to build the software, because nobody would have to sit there and wonder how to put it together, or with what parts. Just follow the plan.