When I do TDD and when I don’t
codewithjason.com
codewithjason.com
On top of this, I've also come to find unit tests to generally require more work and be less useful for identifying regressions than integration tests for the work I do. Maybe this is just a result of the kind of work I do though.
- Write the Test, they're going to fail cause you have no implementation
- Write the implementation, so that those tests are green
- Refactor to clean up the code, make it real nice
Rinse and repeat as needed, until you've settled on a fully tested solutions. If you do this, you are practicing TDD as its intended. I think the overall message around TDD, how it gets talked about in industry, and how its gotten promoted (or not promoted, I guess?) makes it so confusing. This is the heart of it here, is to rinse and repeat these 3 steps, always starting with writing Tests first, which is to validate that you understand what you're building
What I have found in my decade+ doing this is that most of the time when I run into team members who draw blanks or feel that TDD is a roadblock is that's one way or another they don't have enough information about the requirements of the work involved in what they're doing.
Not sure if this helps anyone or not, this has been my general experience and may not capture every case, though I feel confident enough that this is a shared experience that I hope it brings another way to think about TDD in terms of simplicity.
But this is where I falter every time: the test necessarily depends on the implementation. I could write a test for my first pass at a function... but then if I decide I need to actually split that function into two or go for a different approach entirely, then I have to basically scrap that test. And then I've just created a bunch of friction that slows down my development process.. for what gain?
And to your point, yes, it may have to do with the fact that often my requirements are amorphous and I discover them as I go. For example, say a designer wants a specific behavior, but then I realize it doesn't work in an edge case that we will probably hit, and fixing that edge case will take twice as much time. So then I work with them to find a less time consuming compromise -> bam, new implementation, new tests.
Or maybe it's not a user facing behavior, and I realize after 10 hours there's a way simpler way of doing the thing I want. Same thing -> new tests.
Once I've got the basic code layout in a satisfying state, only then do I feel comfortable starting tests and ensuring I most or all of my conditional branches -> success and error cases.
I feel like I'm missing something
Then you're writing a function coupled to the test, not writing logic you can plug into the test to reasonably verify it works as intended.
That's the bit people miss I think. Tests should reflect how the code is consumed, and test for that (inputs and outputs, more or less) not for implementation details. Your test shouldn't care if its 1 function or 7.
If you want to take a more exploratory approach and make up the requirements as you go along, fine, but that won't look anything like TDD.
You can use TDD as part of an iterative specification process as well, but then you need to apply it for each iteration: determine the new specifications, write new tests based on those specifications, and update the code until the tests pass. Then revise your specifications based on feedback and repeat the process.
Mocks vs stubs vs no mocks. Integration vs Unit vs End to End tests. TDD vs no TDD. OOP vs FP.
Grey beards of HN. Where do I learn the best ways to do testing ? I am happy for any kind of pointers. Whether they be books, talks, blog posts or code bases. Anything outside of the usual examples would be doubly welcome.
Well, programming is not a monolith. Different high-level requirements demand different standards and processes, even though for many things a standard solution works. A safety-critical product (or just one that can lose lots of money) demands a different process than a toy app.
Consider Martin's primes kata, at http://www.butunclebob.com/ArticleS.UncleBob.ThePrimeFactors... . The final version is:
public class PrimeFactors {
public static List<Integer> generate(int n) {
List<Integer> primes = new ArrayList<Integer>();
for (int candidate = 2; n > 1; candidate++)
for (; n%candidate == 0; n/=candidate)
primes.add(candidate);
return primes;
}
}
In this case, boundary analysis says that if the function takes an int, then I should include tests for negative numbers, and tests for large values, like 2^31-1, which is MAXINT and also a Mersenne prime.
(Neither of these are in Martin's tests, which only test 1, 2, 3, 4, 6, 8, and 9.)
When should I add the test for 2^31-1? Your "Write the test" says we should only write tests which will fail because it has no implementation, but in this case we expect it to pass because we have an implementation. Do we not write that test?
Which leads to my issue with the "refactor" step of "red-green-refactor."
Suppose you add that test for MAXINT. It takes about 5 seconds to run because of it does >2.1B modulo tests.
Implicitly, TDD tests are a supposed to be fast. Not 5 seconds per test. Or the spec might explicitly require (say) a 1ms execution time, or you might find that system tests fail because this algorithm is too slow.
There are any number of faster factoring methods, as Eratosthenes well knew, so pick one and implement it.
Is this in the "refactor" step? Technically "Substitute Algorithm" is one of Fowler's refactorings, so yes.
But Fowler's refactoring are meant to make things cleaner and easier to understand. Not whole-sale replacements with additional complexity. In the discussions I've seen, the refactor step in red-green-refactor starts and ends with the same tests.
While a more complicated implementation may have its own set of special cases to consider. (For example, a complex sorting method like Timsort needs more tests than quicksort to cover all the code paths.)
The descriptions of "red, green, refactor" TDD I've seen completely ignore these issues of when to add additional tests you expect to pass, and how to refactor for purposes other than "make it real nice."
I agree that starting with MAXINT would immediately 'validate that you understand what you're building.' But most TDD examples seem to start with the easy cases first, not the hardest. They teach an incremental design approach where experience from the easy cases helps progress towards the final solution. My experience is that approach can lead to an implementation which requires a design methodology more powerful than red-green-refactor to resolve.
I was actually surprised what the author was describing was somehow _not_ TDD. I thought it was pretty normal to try some things out, make some decisions, and then return to codifying those decisions in tests.
We do web app development, and 90% of our tests are browser-based integration tests that test whether the given inputs (usually forms) lead to the expected outputs (usually something printed on a web page).
I wonder if this is more natural when you're mostly writing integration tests?
It often helps to start with the design first (even if just a napkin sketch). Once I’ve done that I usually have enough information to write system tests (which also forces me to consider edge cases). Then I can then fill in with unit tests as needed as I write the code to get the system tests for that feature passing.
I dont really get why the practise is so intrinsically linked to the practise of writing unit tests.
Increasingly I'm starting to believe unit tests are a scam, but TDD is going to stick around in one form of another forever.
https://blog.thecodewhisperer.com/permalink/integrated-tests...
I've observed the same, but having the ability to run any given function in isolation is huge for diagnosing problems. Does this function respond correctly to the correct input? Yes? Ok, that's not where the problem is. Next.
However, if you’re still developing the interface definition itself, TDD just wastes time when it transpires the interface you envisaged and wrote tests for doesn’t quite match up with the implementation as you write it.
This sounds like TDD is WAI in this case. If the tests break because of the implementation of the interface, either the tests were wrong (likely because the interface was poorly or incompletely described) or the requirements changed (and broken tests means you had decent coverage). Catching this with a simple test run during initial development is far cheaper than debugging against someone else's expectations in the future.
The problem with test-driven development is that it focuses attention on getting specific features working, rather than finding the best design. This is tactical [as opposed to strategic] programming pure and simple, with all of its disadvantages. Test-driven development is too incremental: at any point in time, it’s tempting to just hack in the next feature to make the next test pass. There’s no obvious time to design, so it’s easy to end up with a mess.
One place where it makes sense to write the tests first is when fixing bugs. Before fixing a bug, write a unit test that fails because of the bug. Then fix the bug and make sure that the unit test now passes. This is the best way to make sure you really have fixed the bug. If you fix the bug before writing the test, it’s possible that the new unit test doesn’t actually trigger the bug, in which case it won’t tell you whether you really fixed the problem.
But it just goes to show any technique (including TDD) is not going to force you into good habits. This needs to be learned with experience.
Then when they're at the top they don't know what better would have looked like ;)
They get to be happy the whole time though.
Design time is the refactoring after your tests pass.
That's when you can move your code around with confidence, since you can lean on your tests.
It's hard to overstate how common this is.
If your design mistakes are in your code and your tests, they'll also likely be in your mental model and your documentation (if it exists). At which point you what? Throw it all away and start over? In extreme cases, but in others you do partial rewrites and refactors. Address the issues.
But not having tests because they may encode your design mistakes is as foolish as not having documentation or comments because they may also encode your design mistakes. In a few years, you'll have a blob of code that, in the best case, is perfectly readable and comprehensible. But, in reality, is likely to fail at communicating the overall design intent and requirements.
And so what if it doubles the cost of fixing the mistake? You made a mistake and it had to be fixed anyways. In the end, I've never found tests to cost more than they saved. A lack of tests has always led to higher development costs (primarily as measured in time to delivery). Regressions are ludicrously common without tests, which eats away at your time (and therefore adds to your costs) very quickly.
Kinda, yeah. Or huge chunks of it. Thats what the OP said. I've done a lot of that.
Was that a rhetorical question?
>And so what if it doubles the cost of fixing the mistake? You made a mistake and it had to be fixed anyways.
Well, then assuming you need 1 month to do a product iteration and 1 to do tests and docs on it and you need 5 iterations before landing on the "right" requirements to meet product-market fit, that requires 10 months to get to fully tested code doing the right thing with TDD and 6 without.
What if your runway is 8?
>In a few years, you'll have a blob of code that, in the best case, is perfectly readable and comprehensible. But, in reality, is likely to fail at communicating the overall design intent and requirements.
I've been on both sides of this problem and I honestly think that this is much less dangerous than not iterating on requirements fast enough.
I've bailed a company out of a massive technical debt hangover before. It's horrendous but it's not usually fatal. But solving the wrong problem? Not finding the right problem soon enough? That's usually fatal.
What you are describing is not refactoring: "In computer programming and software design, code refactoring is the process of restructuring existing computer code—changing the factoring—without changing its external behavior."[0] The interface is an integral part of external behavior.
With that said, what is "internal" vs. "external" behavior is… fluid. A large-scale refactoring might involve changes to internal components with their own interfaces and tests. The tests for the component being refactored should not be affected, but in the course of refactoring the larger component you might make more extensive changes—not just refactoring—to the smaller pieces making up its internal architecture. For that you would need to design new interfaces & requirements for the internal components, write new tests, and then iterate on the implementation until the tests pass, just as with any other TDD process. In the meantime you have the unmodified tests for the larger component to verify that your new internal architecture still satisfies those higher-level requirements.
Sure, in with bigger refactorings, some unit tests probably change. That's just a (small) part of everyday work, as I do it.
Adjacent parts of our company were working on a Heroku clone PaaS, and I got some time on a team building an auto scaler service to spin up and down other services. We didn’t know how our service would kill and start other services. Tests were changing as much as production code while we worked on that. A lot of our work was spiking to see how our service fit into the ecosystem of other services.
Dogma is the bain of everyone :) I will never advocate "100% TDD." It is a tool like any other that has its place and, more importantly, doesn't fit everywhere. Though I will freely admit I do encourage people to use it more than what I have seen to be the typical amount.
> When it came to adding a feature to a Rails app, the interface is pretty well defined, it’s going to serve up HTML and js, it’s got to fit on the existing structure of the page or conform to a design.
It's very unlikely I'd use TDD on that. Boilerplate/plumbing is something I rarely use/advocate TDD for - it's the "clever" stuff you want to cover. The stuff that is following rules outside of the code itself.
> Tests were changing as much as production code while we worked on that.
Absolutely nothing wrong with that, imo.
When your requirements change, and you change your code, you have two questions. First, did my code change do what is needed for the requirements change? You modify tests (or write new ones) to answer that.
Second, did I break anything in the process? The remaining tests answer that.
Don't productionise prototypes.
But that should be part of the "production-izing" process, right? You convert it to a solid design instead of a quick-and-dirty hack?
The whole point was getting to a design that fulfills the feature requirements but for which the contract boundaries weren't known going in.
It's frequently scorned but Im not so sure it's always the worst idea in the world.
I've built shoddy hacked together systems that made customers happy and wasted months building heavily tested well structured code doing something nobody wants.
Well engineered tests take a lot of time to build - sometimes 2x the code itself. If you've built the wrong thing youve paid 3x the cost to figure it out before going back to the drawing board.
I'm a big believer in retrofitting tests once code has proven itself useful.
Why? Because if I find it hard to write the test, it's telling me that I'm likely to find it hard to write non-test code that uses the class/module/subsystem/whatever. It forces you to use the interface to your code. If it's hard to use, that's telling you to consider changing the design.
No design gets to the coding stage until someone has answered "how will this be tested?"
So, yeah, you should absolutely be designing around tests.
The design spec documents determine what the implementation should look like, and the tests should verify the implementation works as desired, not drive its structure.
I don't want the implementation to be chopped up and scattered everywhere in the name of "testability" because someone decided that a tool should dictate the architecture of the code. That leads to unreadable, hard-to-maintain, inefficient code that, with a tight coupling to the test suite that makes both them of fragile in the presence of changes in the other. For very little gain.
I completely disagree with everything above. Literally, take the sentence and make almost every word the opposite.
> Requirements drive your tests, which drive your code.
It's verification of implementation, not a unit testing, so it's a VDD, not a TDD.
It's in the name.
Sure, just writing tests is entirely about automation, but TDD is more than just writing tests.
Originally, it's Test Driven Development.
I see no connection between TDD and (good) design, but other folks see.
[1] https://www.goodreads.com/book/show/387190.Test_Driven_Devel...
Genuinely curious, since I generally agree with what they said. However, I've been recommended that book before with the same sentiment and am quite curious about opposing opinions, just haven't found the time to read it yet.
Will I use TDD for basic things? No. I'll very probably write tests for them though just to provide regression coverage.
I always feel like I should be doing more tests. That this is the correct way to do things.
Being a data/web analyst I have to say that I once created extensive tests for a specific part of custom javascript logic setup by a former agency of my client in the tag management solution they used.
I even had to mock basic functionality from the tag manager (Adobe Launch) to make it work locally.
When I took over it was a mess of code that mapped url parameters to their specific marketing channel logic (you could have done this purely in Adobe Analytics, though - but I was never able to find a way to explain the unnecessary complexity of the implemented solution).
In the end I had created around 60 test cases to ensure this fragile bit was in a way that enabled refactoring. It worked.
Especially when around one year later, a colleague of mine who had taken over the client needed to change stuff in there, had it break and asked me. I thought his solution should work, but the tests told a different story. Within half an hour we had it nailed down, fixed and running flawlessly.
If I ever have such a complex piece of logic anywhere I will surely learn how to write tests in the respective language of choice. Until then I will happily fatfingered stuff on my own amateur projects, though.
( I think the last time I did this was for a point of sale terminal I was working on, which needed a solution similar to the change making problem https://en.m.wikipedia.org/wiki/Change-making_problem )
Anyway, for those situations, I write a large number of tests cases, covering every reasonable scenario, plus a bunch of unreasonable scenarios.
Then I write a half-assed implementation that fails on several tests, and I keep hacking about until more of the tests pass. Once they all pass, I stop. Even if at that point I have no idea why that particular version of the code completely works.
It's nasty I know, but sometimes it's the quickest way to a robust implementation
What I've learnt over the years is that tests lock your code in making it hard to adapt (you could say making you less agile).
When I'm smashing out new code and I haven't established patterns then it doesn't make sense to lock that new code in.
Once I do have a pattern going then I might add tests, but it's not for the sake of test coverage or to gloat that I have tests, it's purely because I actually care about making sure those bits of the code work as intended and that the edge cases are covered.
For these tests I do indeed do TDD.
Otherwise it's a trade-off - would I rather be closer to a feature people can use or would I rather be slogging away writing and maintaining tests.
This being said, I will say that when I come to a point of adding more people to the team tests will become much more important.
The way I see it though, not having tests and having people complaining about it means that I've done something right.
This reminded me of when Ron Jeffries tried to use TDD to write a Sudoku solver, but didn't manage to do it. The take-home lesson was summarized at https://www.infoq.com/news/2007/05/tdd-sudoku/ as "while TDD may not be the best tool for inventing new algorithms, it may very well be the best tool for applying those algorithms to the problem at hand."
In this context, I think of "crisply defined" as being when know the algorithms, and input/output format, and want to get things working together.
I still don't find TDD useful. Even in that case, I much more a spike-then-stabilize developer, with most of my tests added at the end, followed by coverage analysis to identify missing tests.
(And I don't believe for a minute the claims of TDD supporters that TDD naturally leads to 100% coverage.)
I agree with you, but at the risk of No True Scotsman'ing this, one of the TDD tenets I'm familiar with is that you write the least amount of code necessary to pass a test. So if you do it The Right Way, then you should be at 100% or something very very close to that.
Here's the clearest example. For a while I had a code base which supported both Python 2 and Python 3. I dropped Python 2 support and removed the compatibility layer.
I'm still finding places where I have Python 2 code paths. (Clearly I'm not using enough coverage testing.)
Here's another example. Suppose I have a single public entry point, which internally calls a number of private units. I've carefully tested only at that public entry point so my test coupling isn't an issue.
I then realize that I can special case (say) n=0 early in the public entry point. Now, a number of the private units no longer need to handle the n=0 case.
Manual inspection during refactoring might catch all of those code paths which are no longer used. I know I'm not diligent enough to find all those cases. I've even had times where I find an entire function is no longer being called at all, because a refactoring removed the need to have it.
Once you've had a few refactorings of a non-trivial code base, even during its greenfield development phase, you're almost certainly going to have dead code. Even when using TDD.
No, I don't know what the coverage rate would be on a TDD project which doesn't use code coverage.
But my experience tells me not to believe Kent Beck's statement "TDD followed religiously should result in 100% statement coverage".
The basic idea is: read the specs, write the tests, then write the code that passes the tests. The big problem here is that you add a level of indirection, and something may be lost in translation. That is, if the tests are wrong, then your code will be wrong.
Since you already have the tests when you start coding, there is a tendency to forget the specs and code the tests, since your goal now is to have "all green". And because if you did it well, it is very easy to test your code, it is tempting to just tweak it until it passes, without you understanding what you are doing. I know because that's the process that lead me to the first mentioned terrible code.
For a simple analogy, TDD would be like giving students the test subject before they start studying. Sure, they will get good grades, but don't count on them studying what is not on the test, and some will simply memorize the answers. That's why at school, you almost never have the test in advance.
Obviously, I am not against testing, especially automated unit testing, but the core idea of TDD, that is writing the test first, is, I think, a bad idea. The often cited argument is that because you start with a failing test (because you haven't implemented the feature yet), you are guaranteed to actually test something, but I find it to be of limited value, except maybe on very large projets where not implementing a feature at all is something that can happen accidentally.
An approach that I think works better is to write the code, write the tests, pass the tests, intentionally break your code, check that the tests fail, revert your changes. Code and tests can be written at the same time, ideally by different people, but if you do thing in order, I would go code first.
In fact, I think the only real "advantage" of TDD is that it simply forces you to write tests. It is an anti-corner-cutting technique.
The value from TDD is documenting those assumptions. "The db will always be connected before this class is used", "This function assumes the user is authenticated" and so on.
Then in 6 months, a year or even 5, when you or someone else comes back to this code, you can be reasonably certain changes to it will break tests, because the tests tell you how the REST of the codebase is using this.
Of course, that assumes a decent shelf life for the code. If you can be reasonably certain you're going to handover this and never touch it again, don't bother with the tests. And delete the repo while you're at it.... (not comfortable with that, are we? :D )
The above assumes though that I’ve done enough previous discovery to have a relatively robust model of the problem domain. If not, I prototype first and during this stage I won’t be using TDD. Prototypes produced are throwaway, of course.
If the user is a machine, use a machine to test the code. If the user is a human, use a human.
The user does not know or care about unit tests.
The state space of any non-trival algorithm is far greater than can be sensibly enumerated.
Software is design - designing something inside out makes no sense.
When I write the tests first, it gives me a feel for how consumers of the API will use it. It's a great method to understand the UX of the re-usable code.
> A test is the first user of your code
[1] The Pragmatic Programmer 20th Anniversary Edition (Thomas, Hunt)
I expand on this specific point in a blog post: https://www.buildthestage.com/just-enough-automated-tests/
Test first design makes a lot of sense, but when you’re unsure about the design using code to figure it out seems just as reasonable as any other method.
If not what’s left? Do everything waterfall?
What I was feeling and trying to convey is how it flips the design process from bottom up to top down, by forcing the design of the interface first. From my experience you miss a lot by doing that. The reason is when you start from the code interface based on some requirements you most often find an impedance mismatch when you reach the bottom AND you don’t get the chance to tweak or rethink the requirements from straight, simple bottom-up code. Requirements being natural language and by nature informal and incomplete (else they would be code), building on them is risky. Building on the software APIs you stand on (starting at the bottom) is much less prone to change. This is the sound approach in my opinion from an engineering perspective. You start from what exists and what you stand on and grow you software by aiming for the requirements, trying to land the closer you can to them.
This process of discovering natural interfaces that emerge when you specialize a lower software layer API is the most profound and impactful realisation (and huge boost in both productivity and quality) I had in 25 years of programming.
Everything starts to fit, have easy path forwards, becomes easy to maintain and evolve.
I think the gut feeling I have when I see people advocating for TDD is that. I feel it will prevent proper co-evolution of code and requirements, and lead to the usual mess.
https://en.wikipedia.org/wiki/Telecommunications_device_for_...
Use your imagination some time.