The Big TDD Misunderstanding

I understand the point you're making. But, I find this retort to be every bit as frustrating as the rhetoric you're criticizing. It gives us permission to dismiss things we don't fully understand or that require experience and practice to master as snake oil.

Surely most of us would struggle to pick up general relativity or neurosurgery even after weeks or months of training. But we're not--I hope--going to dismiss our neurosurgeon instructor when they say we were cutting something incorrectly even though we REALLY THOUGHT we were doing it right this time.

Maybe TDD really is awesome. And maybe, simultaneously, it's not easy to write a 100% prescriptive guide for how to apply it to every kind of project.

I feel similarly about the rhetoric around OOP. Nine times out of ten, someone complains about OOP and cites some kind of Poodle-Dog-Animal class hierarchy. Then someone comes and says that program object relationships aren't really supposed to be taxonomical. Instead of taking that to heart and wondering if maybe they can try OOP again with a different mindset, the response is defensive. "Surely OOP is still terrible because I was taught the wrong way. And if it's possible to employ a technique ineffectively, then it must be a terrible technique. OOP sucks and they're in a cult so they can't admit it."

Do you have any idea how many wrong ways there are to use the controls in an automobile? Yet we still mostly blame the drivers when they cause a collision because they were using it wrong.

For what it's worth, I think the article is right about the evolution of the term "unit test" and the author mentions the "classicist" approach to unit testing, which is really the one that makes sense with TDD. The "mockist" approach to unit testing, is "simpler" because it just makes an individual class or function the "unit", but that will make tests brittle and make a test-driven approach much more verbose and cumbersome. It also so happens that the mockist approach is the default approach in today's programming languages, testing-frameworks, IDEs, etc.

I often tell my colleagues that if technologies or methodologies are widely misunderstood, then practitioners aren’t to blame.

TDD might be great, but I have yet to see it widely succeed because its adoption is troublesome. It’s a bit like any of these newest products that promise to address everything consumers demand, only to fail miserably against the same old, leaving the few adopters asking themselves why commoners didn’t get it.

This applies specially to business people.

100% test coverage just means you tested that s*t with all you expected...

...but your software does not live isolated.

Some external system may have a bug, or may inject some unforeseen values, or a solar flare might flip the value of a bit in your system and crash your software.

In many disciplines there are activities that end up feeling like 'scaffolding', learning exercises that you don't necessarily do all the time. They make you a better practitioner the first time you do them, continue to add value for some time, and then are useful from time to time as a refresher.

I think if you've never done TDD, you're missing out. If you had to be dragged kicking and screaming into it, you're missing out (possibly have damaged yourself in the process) You don't need to make a cult out of it (and I suspect some of the resistance to it is from people who feel like they're being asked to join one), anymore than you have to religiously clean your shoes to see value from having a set of cleaning tools.

The same for pair programming. It makes you look at some classes of problem in an new way, and those lessons can stick with you even when you are working alone on a personal project.

Most of the time these days I tend to do a testing sandwich. When I'm still trying to figure out what the APIs will allow I'm free-form writing code trying to stick the concepts together. At some point I get stuck trying to juggle corner cases, and I realize that I'm grinding gears and need to write test code for a while. At this point I still get a lot of the ergonomics benefits of TDD because I'm not yet wedded to my implementation (low sunk cost) - because it doesn't work anyway.

Any technique, in retrospect, could have been applied better, in an improved form.

This. Basically, TDD is exactly the same as BDD, only with a developer-oriented (as opposed to product-oriented), traditional assertion syntax.

There’s tension between testability and good design (encapsulation, achieving more through abstraction, hiding/reducing complexity, usability, etc). Starting the design process from API or method calls makes no sense at all unless one wanted to reanimate a failed testing method.

- When I change the logic of a statement somewhere in the code, at least one test will fail in the suite.

A high % in a test report may give an indication but is far from giving me confidence. Having good code coverage means that, once you are back to green, and you have done to test code what will keep or improve your code coverage confidence, you can send the PR being tranquil that you didn't break anything else.

Have you considered that, because TDD is a design practice (a 'meta-design') for writing code, there is no "one correct" interpretation, or misunderstanding, just different understandings? Yes, the people might be misusing it as per your understanding of it, but you are not the center of the universe here.

You are correct that "it's about X", the people who are misusing it are also correct about their interpretation of it. Fundamentally, all TDD is the 5 steps of "writing a test first, and writing code that makes that test pass", along with some mentions of other meta-design approaches like KISS, etc.

The way someone understands a meta-design is driven by a lot of factors, the way someone uses a meta-design is subject to the cultural context in the place you use it, the design restrictions and constraints of your tooling, the meta-design constraints of what practices management is willing for you to devote time to, etc.

So fundamentally, you cannot create a design practice (or meta-design) that is applicable to all scenarios and circumstances. TDD will be outright inappropriate for some situations. This is exactly what "design patterns" are about, finding similar or identical ways of designing code, categorizing them, and figuring out what circumstances they do and don't work in. Now, I guess we need "design practices patterns", so we as an industry can move on to figuring out what practices are fit for certain scenarios, and what scenarios they are unfit for.

Ultimately, anything less than the understanding that "context is king" is likely to lead to people just going around in circles and talking past each other.

I have found TDD to be an extremely useful tool in cases where I was not sure about the design but knew what the requirements were.

My problem with this is that Tests != Design.

I hear this all the time "Oh, TDD isn't about having 100% coverage it's just making sure developers think about what they make before they make it"

If your developers don't think before they build, fire them

Tests aren't design.

Tests also aren't documentation, which is the other thing I hear a lot.

You write the test before have implemented the authentication system. Either generally entirely, or just in the "delete something" subsystem. You write a test that says when unauthenticated user tries to delete something, an error of the appropriate kind happens. The test fails. Then you implement the authentication system to make the test pass.

I have written this exact kind of test.

I do not universally do "TDD" though, I probably write a failing test before I've written code less than 25% of the time. I do write what you call "negative" tests, sometimes TDD sometimes not -- I think I don't write "negative" tests TDD any less often than any other kind of test. I may not write "negative" tests enough, but it's not because of TDD (since I don't mostly write tests first!).

So I'm not here being a universal TDD adherent. I guess I'm just being boring and unhelpful "it depends, and I just use my intuition built on years of developing." (I think this kind of intuition generally grows after years of developing on the same platform, so people switching tech all teh time doesn't help. But that' anotehr topic).

I am aware "you just need to have skilled developers using their judgement" isn't particularly helpful. The attraction of TDD is the idea that you don't need that, a novice can follow this process and still achieve reliabile and robust results, efficiently. I don't really agree... but also think it's important to know about and have experienced TDD so you will have it in your toolbox to apply sometimes. I don't think "negative tests" is a particularly useful category for deciding when -- I have definitely done TDD usefully with "negative tests", you can totally write a failing test for something the system shoudln't do... in many circumstances.

I agree that TDD in the hands of an inexperienced developer isn't a substitute for stepping back and thinking about design/architecture, you still need to think about design/architecture. But also that TDD has definitely sometimes helped me think about design/architecture -- sometimes in a really profoundly useful way. The very unhelpful "it depends".

Negative tests from what I've seen accrue by encountering unexpected results from code in the wild and writing tests to make sure they are handled. They are a result of code being used, not by proactive or predictive design.

What makes you feel it's "a lot of practitioners"?

My biggest pet peeve in this context are test coverage requirements for OO languages that feature properties or getters and setters. 100% test coverage would require writing "tests" for those, which basically amounts to testing whether the compiler works - madness!

Yes, they should but often they do not. The likelihood to get it right is higher using an outside-in approach.

You should take that as a sign you need to rewrite the code you’ve just written, but a lot of people don’t want to trash all (or part of) their hard work and instead bend the tests so that they deal with the code.

You now have bad code, and you have bad/fragile tests.

I think TDD (when enforced) is mostly meant to prevent people from falling for that.

That said - there is a problem with writing tests last: You don't actually know if your tests are going to fail when they should, because you haven't run them on a code-base that isn't fully implemented.

You can deal with this by stashing the implementation changes and running the tests to confirm they fail where expected, but sometimes this is very challenging because you need a partial implementation to reach some of the failed assertions. This requires thoughtfulness and discipline and so it is hard to maintain this consistently across a team. It isn't possible in a code review to confirm if people have done this, and often it isn't practical to do it yourself as part of reviewing code.

- the time I lost creating bad designed code and manual testing is huge

- all the companies and projects I have joined always have terrible designed/unmaintainable code, made sometime by 10y+ experienced people and explaining at least how to write proper test and how to design a code that is testable is always the game changer to increase a project velocity and down the defect rate

If, for other reasons, I follow the Hexagonal architecture, replacing the PostgresProjectionsRepository with an MemoryProjectionsRepository is both trivial and easier. Replacing an S3LogStore with a StdOutLogStore just as easy and trivial.

But, if, God forbid, I'm stuck with a tangled mess of Rails ActiveRecord Models that break every SOLID principle with both feet, are highly coupled to the database and often even arbitrarily stick business logic in the DB or in the models, then certainly: it is hard.

And while this sounds like a rant on AR (it is!), ActiveRecord has its positive trade-offs, or, often, simply is there. Pragmatism dictates that sometimes the database is hard or impossible to stub and you're far better off just giving up and treating the entire directory of models+the-fully-seeded-database as a single unit instead.

But, what the OP of the article overlooks entirely - and what many haters of TDD miss completely: the tests are yelling important information at us: the design is a mess, there is too much coupling, we lack abstractions, there's too much going on, we are building God-classes and so on.

But, again, sometimes pragmatism dictates we ignore this and go for the bad design anyway. As long as we pro-actively chose to go for the Bad Design (rather than unknowingly grow into it) this is fine, IMO.

Even if it is easy to spin up a container, seed a database inside that container and drop and recreate that database after every single test (so that tests remain a unit and can't influence each other) - it tends to take a few seconds, which instantly makes TDD tedious to do. Thus you start making more changes at once, run the tests less, and drift away from doing TDD...

The most typical solution is to run databases in-memory rather than in a container.

My personal approach though is to make my code more modular, so that only the "database accessor" class needs to mock the actual database, while the "query builder" needs to mock only that database accessor (much easier) - and the database class needs to only mock the query builder (even easier) - and the actual application code needs to only mock the database class (super easy).

The Game example: e2e tests for games probably do not make much sense (I've no experience here). So you will have components/unit tests. Those have a defined interface and behaviour. You want to change this, you change the test first like in any other project.

The Audio engine - what is an audio engine?

If you wrote an algorithmic function crucial to your app, isolate that MF and write a unit test. If you're testing a transformer or API response, you don't need to isolate each layer and run/write your integration tests at this level. If you can't isolate your code for unittest and you're writing OOP your architecture probably needs to be rethought.

That said my goal when writing something new is to have all the critical/core functionality covered on the backend, with a test case for edge cases. The biggest goal is I don't know who will be working on it next, and if they miss-understand what a function is doing because they are just trying to fix a "bug", the test case should catch it. That's why it's import in TTD to have one person write the tests and someone else write the code. More so if your team , is a clusterfuck particularly some of the senior devs are hacks.

That's not a fragile test suite, that's a test suite working well: you want tests to start failing if you introduced a regression to the overall program behaviour.

> the changes to fix that regression causes other regressions

It sounds like what you're really getting at is something different from test fragility: specifically, if a test fails, how easy is it to identify where in the code a fix needs to be applied. That is also an important consideration when writing tests but it's completely separate from test fragility.

> Integration suites have value, but I use them with discretion because I know that they are a challenge to maintain.

This is the opposite of how the test pyramid is supposed to work. Integration tests are supposed to be less fragile because they don't rely on specific implementation details or mocks: they're mimicking a real user or a consumer of your API. If you are getting false positives from your integration tests a lot, then either the tests are written poorly, or you're making too many breaking changes to your library/product/whatever.

> Tight unit tests that test small pieces of logic thoroughly and make ample use of mocks to ensure deterministic outcomes are a must. The code can change all around it but that function will work as advertised until someone changes it. Then they can update its test suite and you're back in the golden zone.

That's completely useless as a test then. Of course if you change code around it the test won't break. The whoe point of a test is that it doesn't break even for (some) changes to the code being tested, and if you always have to update the unit test when you change the code then it is providing no vlaue and giving you a false sense of security that all your tests are green.

Unit tests are great when you have a particular irreducable complexity in a piece of code, or more generally they are good when there is some constraint that should be locally upheld that is likely to be more permanent than the specific implementation.

This is why isolation and encapsulation are important.

Your tests were telling you that A has intricate dependencies on B. If the tests for A start failing when you refactor B, they were (too) tightly coupled. The tests are telling you this (whether or not you act on this is another discussion).

And when the tests for A keep passing even when you broke B -false positives- you should see at least one integration (or even e2e) test failing. If not, the tests are showing a hole in the coverage.

If A does something important/complex/critical/etc maybe the important/complex/critical/etc part can be refactored into a dedicated function A1 without external dependency and then tested on its own? That way, the tests focuses on where the money is and gets easier to work with at the same time.

Only mock the things you can't easily set up in your unit tests to get them to work, like database connections. For spring-based components, try to use the real components where possible, but where needed use mock components that do enough to get the system working.

That does mean that if you are testing function A you are also indirectly testing function B, but that is fine.

Counterquestion: how do you know, at each release, that the paypal-stuff still works?

If the answer is "I test it by making a purchase at each release", the obvious question is: why not automate this? That is one benefit of TDD.

So why not start of with that automation?

In your specific case, I would write a simple test first, that, says "on the page of the orange Y-combinator swag, order a shirt and receive a timely confirmation.".

Which is subtly, yet importantly different from "go to X and click order". On the level of such acceptence tests, trying to keep away from implementation details like "pages", "buttons" and so on, helps with solving your problem too: you write down the intent&outcome, rather than the implementation. Focus on the business value. Solve that first. And fixate the outcome for all future: regardless of the PSP, pages, exact values (timely vs within 1 sec), actual confirmation-channels and whatnot, the business value is "green" when it works, and "red" when it doesn't.

Then I start implementing it. Maybe just add one of these <iframe> things that Paypal offers, even. I don't really care: the quest thing to make it work. as long as it gets me to have that acceptance test green with least work, I'm happy.

Happy, but certainly not done. Because the most overlooked part of TDD is the Refactoring. Red-Green-Refactor. Once it is green, start refactoring. Replace the iframe with some library. Go a step further and replace that with my own paypal-API-client. Connect that to my event-stream, or not.

As soon as you need elaborate setup in order to test a single piece of functionality (or function), you will start to rethink your design. In such cases there's too much coupling, too many dependencies, or too many responsibilities within the function under test.

Seeing tests as separate from implementation and design is a much bigger issue IMHO. TDD only provides value as opposed to overhead, if you understand it as an integral part of the design and implementation process and use them with that in mind.

The idea behind TDD focuses on a bottom-up approach instead - if all the parts behave as expected, it is reasonable to assume that the entire system will.

TDD was also intended as a tool to help identify problems with design early (e.g. too many dependencies, tight coupling, too many responsibilities, reduce/eliminate side effects, etc.), allowing for rapid feedback (tests should execute quickly and require little to no setup).

Both approaches have different goals in mind and serve a different purpose. Ideally you always have both - run unit tests after each code change (i.e. often), run integration tests after each added or changed behaviour (e.g. daily or as part of an acceptance test process).

Starting with integration tests much work up-front and adding unit tests after the fact can introduce considerable overhead for questionable gains.

This other definition of "a test that doesn't depend on other tests" seems of little use, since no tests, even in something like a UI regression, should depend on other tests.