Hacker News Reader

Comment by hacker_9 | Hacker News Reader

I expect the reason TDD is so controversial on here is people can't see the long term benefits of tests, and instead only think in the short term. But in the commercial world, code you write can potentially have a lifespan of 30+ years. In this case, making a choice to write tests is the difference between writing a maintainable component in the future vs writing a soul-destroying 'legacy system'.

If you agree tests are a good idea, but think TDD is too extreme, consider that TDD simply makes sure you write testable code from the outset. When you have a test wrapping a method, and need to add a dependency, you actually decide to use DI (Dependency Injection) because otherwise your current tests will break / become integration tests. TDD makes you think upfront about about things like mocking, separation of concerns, etc.

When you have the mindset that you will absolutely 100% write tests at some point anyway, TDD is actually a faster and more fun way to develop than bolting on tests afterwards.

View on HN

This whole thread is a response to: https://news.ycombinator.com/item?id=15591190

Your points are directly addressed in the pdf. One of his general points being, in practice, the tests become the legacy system instead. And I'd add to that. Given that you've at minimum doubled the code (and doubled the bugs), it seems like a really bad long-term trade off.

Also DI does not reduce coupling. I've seen plenty of code with DI that's just injecting like 30 things, which is obviously therefore coupled. It just makes it really obvious, but DI itself has massive downsides.

If you've ever worked with bad programmers and seen it in the wild now, I'm sure we can agree DI and TDD doesn't stop bad programmers writing bad code. In fact, all it seems to do is make even more of a mess.

Not only do you have to pick apart the bad code, you have to start dealing with carefully moving methods to the right places because DI can make it hard to figure out what's being used where, and then on top of that tests break all over the place because they're entirely dependant on the implementation instead of the functionality.

What exactly is your experience working in a commercial environment? When the updates you ship can affect tens of thousands of customers? In these situations, 'Sods Law' often comes to mind - "what can happen, will happen". If you have a defect in your untested code, you can absolutely bet it will come out at the most painful time possible in front of all your clients. Being blamed for that kind of stuff is a stressful way to live your life, much nicer to have tests shout at you instead.

* and to reply to your edit: of course tests break when you change the code they are testing! But after reviewing the broken tests, you see the intent and re-adjust the test. But what often happens, is you realise you didn't fully understand the code previously, and actually after reading the test you need to undo your refactoring as it didn't make sense in the first place.

I think that everyone should have a 5 year stint with zero tests. I genuinely believe it offered me an insight into the deepest parts of depression.

In saying that, it was raw and I think I enjoyed my job more. There is no safety harness. You write decent working code and sign your name by it or you run home to your mothers nipple.

I definitely learned a lot about programming during that time. To take a terrible system and try to research ways to make it better for your own mental health, not the companies is a personal drive and golden age of discovery which plays a part of my programming to this day.

I get none of this from TTD. It is intensely dull and unsatisfying but gets the job done.

Had over a million people use my code last year, when I checked for vanity. 12 years commercial experience.

I'm effectively the tech lead for two startups, one getting about 750k users per year, the other 250k. Freelancer, but I do a lot of time for 2 clients at the moment.

Both were projects started by other people, but a significant chunk of the code is mine now (one of them I've virtually totally rewritten from VB.Net to C#) and the other I've done huge amounts of refactoring to fix big performance problems, reducing the "main" pages from 10 sec load times for complicated orders to 250ms.

I've also refactored a lot of javascript for both of them without any tests, significantly improving client-side compile times and page load times.

I played with unit tests a few years ago, one of my clients has some, but we mainly don't add any new ones and they never catch anything[1].

[1]That's a slight fib, one caught a bug last month for the first time in the 1.5 years I've worked for this client. It would almost certainly have been caught in testing though.

Pick the tool for the job, if your experience is with things that last 5+ years then ok. I work on a system which is 5 years old and all things from 5 years ago are not relevant. Actually I am working on it only for 3 years now but we pretty much each year rebuild whole thing. With minor things going in and out. Of course we spend a lot of money on automated tests but those tests had to be thrown away because of amount of changes in the system.

So you disagree with TDD, but you need to rebuild your system from scratch every 5 years? I feel like this is an argument for TDD.

As for throwing away tests; Unit tests are meant to be pretty simple - rule of thumb is you can run a thousand tests in ten seconds. Arrange, Act, Assert - they don't need to be complex, they just need to imprint the intent into the codebase. If the intent changes, by all means remove the test.

> So you disagree with TDD, but you need to rebuild your system from scratch every 5 years? I feel like this is an argument for TDD.

Not necessarily. If the rewrite is only addressing issues that would have been prevented with tests, then sure, this is clearly an argument for TDD. However, if the rewrite is going beyond what could have been provided by tests, then having a large testing system could actually make the rewrites harder, which means it's an argument against TDD.

It's hard to say which is the case without knowing the details, and I'd certainly err on the side of saying a yearly rewrite is not a good sign. But if a company is undergoing rapid growth, it's not that unusual for certain systems to be rewritten frequently as fundamental new insights are gathered about how to tackle problems that are hard to scale. And if the system isn't that large to begin with, periodic rewrites could be easier than writing a single version that's supposed to last 10+ years as business requirements dramatically change and expand.

I agree. My experience is:

1. Wow it made 10k! Lets rewrite

2. Wow it made 100k! Lets rewrite

3. Wow it made 1M! Lets rewrite but let's properly think hard about the future of this product (Introduce tests and more people).

ALWAYS DO TESTING should always be given the context (if budget allows).

We rebuild because business needs change, because compliance and stuff. It is not like we rewrite because code is crap. We have like 30 percent of test coverage because that is part that changed only a bit.

This matches much of my experience with DI on real projects. Dependency injection is, unfortunately, used to reduce the labor required to achieve massive coupling. Of course the intended and potentially useful application of DI is being able to rewire an application with different components for different purposes or different environments, and the tradeoff for this flexibility is that you lose explicitness. You can no longer see the explicit wiring in code, which is a big downside. Yet I've worked on several DI-heavy projects with seasoned engineers who get this completely backwards. They see DI as a labor-saving device that allows them to write heavily coupled systems without having to explicitly work out the dependencies between things. In fact, they even see DI as eliminating the cost of complex interdependencies because it reduces the work and cleverness required to create them.

Partly I think this reflects a desire to work on "real" "enterprise" systems. Instead of fighting to keep complexity below the level where DI actually helps, they embrace the flattering thought of, hey, we're doing big boy work here — it's going to get complex, so we'd better use DI from the outset. People who take a lot of pride in working on big systems can't help creating them.

When TDD first started coming out years ago, in an email thread I asked one of the proponents how they handle the fact that tests can easily increase the complexity of a system by adding more code. I was expecting a rational discussion of how to balance tests making your code simpler with tests be a source of added complexity. Instead it was met with anger because I should "count" tests as part of the codebase, and how stupid I was for even considering such a thing.

I still stand by my point. 100 lines of code with 1000 lines of testing code means your codebase is 10x what it would be without the tests. That extra 10x had better result in a hell of a lot of benefit.

Additionally, over the years I've found that easily-tested code sacrifices readability, and that tiny functions isn't always the best way to go. I've gone back to liking larger, more monolithic functions as often I don't want to create a bunch of generic functions for hypothetical use for other projects, I just want some code that fits my needs to a T.

> I asked one of the proponents how they handle the fact that tests can easily increase the complexity of a system by adding more code.

Adding more code != more complexity. Complexity comes from high coupling between code, that doesn't separate concerns, which make it difficult to untangle, aka. spaghetti. Unit tests are mean to be simple, and anything they test will be no more complicated that how the system will actually be used. In this way they clarify the code, by encoding the intent, available for all to see for years to come.

> I've gone back to liking larger, more monolithic functions as often I don't want to create a bunch of generic functions for hypothetical use for other projects, I just want some code that fits my needs to a T.

This isn't what TDD causes, it just makes you break down things into small testable units. It doesn't make you create super generic wrappers, which I also agree are useless because of their un-readability.

Again, this is the argument made to me over 10 years ago, and again, they are wrong. While it is possible to add more code and not get more complexity, it is very difficult. Generally speaking, adding code adds complexity. Full stop. Excellently executed TDD will reduce overall complexity, but I almost never see that level. Rather, TDD's ability to catch bugs is mostly enough to reduce bugs (and other benefits) overall. But the costs are high.

Tests make clear the intent of the code, how else can you get that exactly? And the next time you have to fix a broken unit test, think of how much time it has saved you by constantly running autonomously as part of your build process.

> Given that you've at minimum doubled the code (and doubled the bugs), it seems like a really bad long-term trade off.

I'd say you've at maximum doubled the code. The test ensures you write only what you need to get the test pass. Without them, devs get distracted and wander until the feature works. Usually distracting themselves with tons of YAGNI violations along the way. In my experience, untested code bases have a ton of unnecessary code.

I don't understand how it would double the bugs. The article has references saying it reduces them. But, even thinking about it, I don't see why you'd say that.

That article is garbage. For example, the claim that an object has trillions of states ignores our ability to classify ranges of values. Sure, an int may contain "four billion states", but for the requirements (which he makes such a big deal about) its highly likely that we can classify the integer into three states: less than zero, zero, greater than zero. As a bank, I might not care how much money you have, only that you have more than zero. In a transaction, I don't care how much money changes hands, as long as no money is lost. The article is a tantrum.

> we can classify the integer into three states: less than zero, zero, greater than zero

Err, not quite. We're running on computers, remember. A comprehensive unit test that doesn't go through every possible integer input should still include:

maximum and minimum integer values for that size

maximum and minimum expected inputs

0, 1, -1

So, there's 7 test cases for a single integral input. And since you bring up banking, let's imagine you're working with a function with two inputs. Since most bugs come about because of the interaction between two variables [0], you'd want to check out each combination. So, 7 possible inputs for the first integer, 7 for the second: 49 test cases.

Why bother? You're working for a bank... imagine facing a client whose balance shot negative because of an overflow error and started accruing "overdraft coverage" fees.

Back to the original point: when was the last time anybody but the AFL tool wrote the "proper" 49+ unit tests for an "add_to_balance(int, int)" style function, when one test would give you 100% coverage for that function?

[0] https://csrc.nist.gov/Projects/Automated-Combinatorial-Testi...

If you mean "nobody would write 49 tests for a function when they could get away with writing one test and have it show up as 100% code coverage because people are lazy bastards" then I agree, and that's the problem: cargo culting the metrics. Look we have 100% code coverage! We are awesome!

If you mean, "no look seriously, you'd need 49 tests for every method because you're a bank", well then, no, you actually need less than 49 (since some are subsets of the others), and in reality even this would get very tiring so you'd write a currency class that can't overflow, and yes, being a bank we would test every case and certainly use automated test generation while we're at it.

But if you mean that in general, for non-banks, the classes of values is too large to manually write unit tests for, then we disagree. That means that your method is too complex.

> some are subsets of the others

In a purely mathematical sense, yes, this is true. But with computers those subsets matter. For example, I wouldn't drop the "max expected" for the "maxint" test; an overflow exception or implicit type change in the case of the latter shouldn't be tolerated for the former.

> That means that your method is too complex.

Would you consider the Python `requests` library's `get` method to be too complex? It has 10 parameters to it: 4 strings, 4 dictionaries, 1 file-like object, and 1 tuple/AuthHandler option.

If you conservatively say there are 5 test cases which should be tested for each parameter (a very low count, especially with strings and dicts), there's approximately 1,125 test cases that should be explored.

Not quite a trillion, but it's still a lot.

Without getting into the pros and cons of python (which will likely get me downvoted to oblivion), 'get' is an example of what I would call a coordination method.

I coach my teams to think of two kinds of methods: methods that do data processing, and methods that coordinate. To test a coordination method, you don't need to care about the value of the things it has to deal with. Maybe you care the they are not null, if your language has nulls. But other than that, what you are testing is that if your method A is supposed to pass arguments 1 thru 5 to method B, and 4 thru 10 to method C, then that's what you test. You don't need to test that argument 1 is a positive integer. The test for method B will do that, if it needs to. And finally you might test that if we expect the result of method B to be passed to method C, the test for our method A would do that using a mock B. Again, what mock B returned could be any object: we only care that it got passed to C.

So it looks like its 1,125, but its actually three.

If we have classes that are SOLID, then we tend to see the processing methods and the coordination methods end up in different classes. sessions.py isn't a great example of SOLID, and I would have a hard time writing tests for it. It is certainly not the kind of code you get if it was written with TDD.

I forgot who said it but it goes something like, "any code without unit tests is legacy code."

Which by definition means your tests are legacy code. It's turtles all the way down. For what it's worth I do think Unit Tests provide value.

Nah, the code tests the tests just as the tests test the code. If you expect your tests to pass and they don't, something's wrong. (Either with the code, or with the tests.) Same goes for when you expect the tests to fail and they don't.

The code is definitely NOT the test of the tests. It may seem that way because (sometimes) when the code breaks (e.g. fails to run at all), the tests also break.

But consider this: what if your tests are poorly written and fail to detect bugs? What if they fail because the test code is buggy? Failing to detect bugs is a bug (undesirable behavior/output) of test code. There are some techniques to address this, for example mutation testing ( https://en.wikipedia.org/wiki/Mutation_testing ), which effectively become some part of "the tests of the tests".

The take away: no, the code is not enough. You need to test your tests.

> what if your tests are poorly written

This seems like tautological reasoning. Naturally, if you write bad tests then they won't be effective, but that seems like a poor argument against testing. That's like saying "Yes, bypass surgery could save your life, but consider this: what if the surgeon is reckless and kills you on the operating table?". Obviously, there is a minimum expectation that the surgeon will follow proper medical procedures when operating.

If it seems like a poor argument against testing, that's because it's not meant to be one! It's meant to be an argument against the notion that "production code tests the tests". It's also an argument for making sure your tests are effective.

> Naturally, if you write bad tests then they won't be effective

But how can you tell if your tests are effective, i.e. if you wrote "good tests"? Production code alone is not enough; you cannot tell if a test is green because everything is ok or because it's buggy/incomplete. You must introduce some degree of quality control in your test code.

I missed this before:

> what if the surgeon is reckless and kills you on the operating table?

Excellent question! What's this surgeon's track record? How many patients have died on her operating table? How up to date with current medical research is she? Is she daring and reckless, or does she play it conservatively safe?

Note that mutation testing is mostly just an advanced (and more brittle) form of code coverage. The flip side is that code coverage reports can also be considered tests of your tests. Not exhaustive, but I'd argue that unit tests hardly ever are as well - and that's OK.

I disagree.

Coverage tests how much of your code is exercised by your tests. Mutation testing is (one way) of testing how effective your tests are -- i.e. how sensitive they are to bugs in your code. It follows that good tests must have both good coverage and good effectiveness, but the two are not the same.

The former seems to me like a (less comprehensive but also less brittle) way of doing the latter. Higher coverage usually means that your tests are more sensitive to bugs in your code - i.e. if a bug occurs in uncovered code, your tests won't catch it.

Of course they're not literally the same, but both are ways of measuring how good your tests are at catching bugs in your code.

It might sound pedantic, but this way of thinking shows you that it's not turtles all the way down: if they'd really be "tests of the tests", then you might also want "tests of the tests of the tests". And since we obviously don't want to keep doing that, you might conclude that we don't need "tests of our tests" as well.

But since they're simply measurements of test quality, we can see that both test coverage and mutation testing can be useful and not a never-ending story.

I disagree. Coverage and mutation measure completely different things, one is not "a way of doing" the other:

- Coverage measures how much of your code is exercised by the tests. It does NOT measure test effectiveness (it's not like mutation, only "less/more brittle"). This can be trivially shown by writing tests that exercise all of your code but have no asserts (or trivial asserts such as 1 == 1). Unsurprisingly, this kind of obviously ineffective test code is often find in the wild (mostly written by junior devs), but less obvious cases of ineffective tests are also common, such as failing to test border conditions. This happens and it's more common than we'd like.

- Mutation testing is a way (but not the only one!) to measure how effective your tests are at actually finding bugs. That this technique exists shows that there is indeed a need to "test the tests", i.e. measure test quality & effectiveness. Production code is NOT "the test of the tests", as someone up this thread erroneously said.

This is like measuring altitude and airspeed: yes, they both measure something useful about your airplane, but they measure different things!

I confess I did not understand the rest of your post.

I didn't mean coverage measurement is a way of doing mutation testing, I meant that measuring how much of your code is exercised by your tests is a way of measuring test effectiveness.

Yes, it can be inaccurate, as your trivial example shows, but it does guide you towards unhandled test cases (i.e. where your tests are ineffective). Likewise, mutation testing guides you towards unhandled test cases (including ones that coverage can't detect).

> I meant that measuring how much of your code is exercised by your tests is a way of measuring test effectiveness.

The problem is that coverage alone is a very poor way of measuring effectiveness, which is why other techniques -- including, but not limited to, mutation testing -- are needed. This is nothing new: limitations of coverage are well known in software engineering.

The code is the “test” for your tests ;)

The code and the test are probably to some degree based on the same logical construct. If that logical construct is malformed your tests may pass and your code might not work.

Regardless if your code works your tests could still fail, likewise your code could not work and your tests pass so since the tests and code may vary independently if either is buggy I don't see how this can possibly be true.

If the logical construct is flawed, no amount of tests will ever catch it, only user testing would.

The point I’m trying to make is that the verification of correctness is bidirectional. If I’m writing tests around existing code, I rely on the code to test my expectations. If I’m writing new code, I write tests to assert my new assumptions. All automated tests do are assert that I’ve written the same logic twice. Writing it a third time will also decrease the likelihood of transcription error again, but at diminished returns.

I believe that was Michael Feathers in Working Effectively with Legacy Code

I've heard it attributed to Michael Feathers.

There are undoubtedly multiple reasons. One reason is that there are domains where the hard problems are integration problems. If the point of your program is to poke/sample some physical object, talk to another opaque chassis/address space, execute realtime tasks (which in practice often includes UI/UX concerns), etc., then extensive integration testing is absolutely vital. If you focus on some other testing discipline and try to de-emphasize integration, there is a considerable risk of fooling yourself about whether your program actually does what it says on the tin.

Yeah I didn't mean integration tests were bad, just that you don't want to turn your unit tests into them unintentionally. Following the TDD mantra, you should absolutely have a failing integration test before writing integration code.

>I expect the reason TDD is so controversial on here is people can't see the long term benefits of tests, and instead only think in the short term.

There's a number of reasons why you could try TDD and get frustrated with it because it ends up sucking for you:

* You're using unit tests where integration tests would be more appropriate.

* Your integration tests would take longer to build than the thing you're building itself because you have to set up some sort of elaborate mocking.

* Time is of the essence but quality isn't / your code will be run a limited number of times.

The code you write can also potentially have a lifespan of 3 months, and it can be hard to predict which it's going to be.

I feel that TDD is an attempt to impose engineering discipline onto something which is still largely at the craftsman stage. As an engineer I like the idea of pushing the field forward, but I don't think software development tools are ready yet to make TDD and the like broadly accepted practice. That means you need to decide case-by-case if they make sense.

Tests define intent. Usually when you craft something, you have an idea of what you want going in. You may even draw it, and in the case of construction projects, fully 3d render it.

Tests are the programmer equivalent, they define a contract you expect from your functions, which can then help you shape the function itself.

> you actually decide to use DI

Whats DI?

Dependency injection

Reply on news.ycombinator.com