Hacker News Reader

Does your team work on a codebase that uses a dynamically typed language? If so then the 100% coverage requirement is perfectly reasonable and arguably the only sane way to develop anything more than a tiny project. If a statically typed language is being used then sure, that requirement is counterproductive but if a team uses a dynamically typed language I don't see a way to have any reasonable amount of confidence that the code is even syntactically and semantically valid without 100% coverage. The 100% coverage is just a tax that you need to pay in dynamically typed codebases (hence my preference to use statically typed language for any reasonably sized work related project).

For 3) TDD has nothing to with test coverage or aiming for high coverage. The fact that TDD evangelists have tried to couple "good testing" with TDD is a huge scam. I don't care when you write your tests as long as they get written before the code is merged in. TDD is good for parts of the code which are very pure or where you know exactly what you are doing, but if what you are doing is more exploratory in nature or is not pure and has lots of external dependencies TDD is simply not practical for a lot of people.

In my experience, scaling a large project in a dynamically typed language with 100% unit test coverage just gets you production outages with all unit tests passing. A reasonable threshold is probably closer to 70-90% _plus integration tests_.

Hopefully you also picked a language where a bigco has created tooling that lets you incrementally add static types -- TypeScript, Closure Compiler, Python typing and/or pytype, etc.

Lol, this is so true. A project that I had joined (late) some years ago had 98%! code coverage, full sonar/fortify code compliance and was an utter piece of garbage that had regular outages in production.

It taught me a couple of lessons:

1. Integration and System Tests are FAR more valuable than unit tests. By several orders of magnitude.

2. Performance Tests are vital. This is actually more true in a cloud world. There is only so much hardware you can throw at a problem before your costs are un-feasible.

This is false. 100% UT in a dynamically typed language like JS is no assurance of anything really. You need to cover all the observable behavior in any case (no matter the type of language) to get any kind of assurance. UT cannot deliver this.

The law of diminishing returns for sure applies here

It applies everywhere.

Also, this kind of mentality frequently results in what I consider negative-value tests. A test suite is supposed to do two things:

  a) identify bugs by failing on incorrect behavior, and 
  b) enable refactoring by continuing to pass when underlying code doesn't change behavior

Focusing on 100% unit test coverage all the time—particularly in code that isn't doing simple, single responsibility stuff—often results (in my experience) in tests that overly mock other parts of the system. The net result is that you end up writing tests that simple check that "the code was written the way it is currently written". Tests like this fail on the first count because they fundamentally cannot detect incorrect behavior, and they fail on the second count because a non-trivial refactor inevitably changes the implementation even if the resulting behavior stays the same.

Not that mocks aren't useful in some scenarios, but if you find yourself writing entire test suites based on them you're likely creating tests that are negative value. They won't catch undiscovered bugs, and they'll give you bad information when you're refactoring.

That's precisely what I did in the beginning of my career. To test every method and mock all the dependencies. This had the impact, because we usually don't use a dependency injection framework, that I had to add an interface for almost every class, adding factories with interfaces almost everywhere and then write tests which expect certain method calls and to basically debug the tests while writing them until all calls have been mocked. Thus, whenever I changed the production code I had to change the test as well.

IMHO in my own projects I tend to write tests which test certain parts of a system as well as some dependencies. So usually I'd say they are integration tests instead of unit tests. But I can refactor stuff and once I break something it's usually detected by one of the tests. Maybe unit tests should test for certain cases border cases, which are hard to test with integration tests.

I heartily agree with your general sentiment, but think you're not mentioning a very important function of a test suite which is:

c) explicitly and accurately document expected behavior including in a variety of edge cases

...this is sort of implicit in your first 2 but I think worth breaking out.

Going for high test coverage (in places where I've done TDD-style dev) has really forced me to think through those edge cases and what I want my code to do in all of them.

I think negative-value testing comes from the mindset of "how do I write a test which exercises this branch?" but a positive version of that is "I put this branch to handle the case where foo is getting close to the boundary. Let's put in test cases with foo at and close to the boundary and make sure it does what it is supposed to." If the test focuses on behavior then you find the bugs and enable refactoring and you get high coverage because if you can't exercise all the paths probably some of them are dead and can just get deleted.

That's the thing that pisses me off about how testing and the periphery around testing (like coverage) are commonly treated. They're seen as being virtuous in and of themselves and no one wants to measure whether they are counterproductive. Even if one took that initiative, good luck getting anyone to care.

Sure, I suppose one could say that any tests is better than no tests, but that also dismisses whether testing can be done better.

Code coverage is the worst. I've worked with codebases with various degrees of code coverage, and I've never seen a correlation between coverage and the number of bugs encountered. There may be an asymptotic nature to how much coverage is needed, but I would bet the threshold is low in a general sense. Once you've reached even 25% coverage, it's really dubious whether more coverage than that is actually telling you anything.

Of course, we could measure success by number of bugs and company revenue, but no, we can't do that; those could make people look bad.

Worse still, I feel like doing unit testing at the expense of integration testing can create a false sense of security.

Unit tests often skip the error cases that are really important for real life scenarios.

We've recently introduced a regression test suite for our data pipeline which simply compares all the data before with the data after running the pipeline. This gave us about 65% code coverage and the certainty that we're testing what actually matters for the product instead of how somebody decides to implement a function.

False sense of security sold by conference indsutrial complex is my biggest issues with unit testing. One time I've seen a junior dev wrote copious amount of unit tests to achieve near 100% code coverage. When this "perfect" code went into prod it almost took the whole system down. Because the junior dev was joining close to a dozen tables and each one of the tables has at least a million rows.

If the junior dev took the time to understand basic DB performance instead of writing dozens of unit tests, then it wouldn't have happened. I blame the gurus who don't write code for living, but peddle magic solutions like "clean code", unit testing etc. to sell more books and seminars.

I'm pretty sure if he didn't write the unit tests it would have equally failed. There is something to blame here, but not the unit tests.

No doubt the junior dev would benefit a lot more from only getting up to 60-70% code coverage.

The extra time could then be used to learn basic Db performance, which is hugely important when tables have millions or billions of rows.

Stuff like indexes and isolation levels should be taught to all junior devs, it's just too risky not to.

I feel the opposite way, with unit tests you can test failure scenarios as easily as any other scenario. So it makes it simple to exhaustively test every code path including failure scenarios that would be very difficult to set up in integration tests.

There are bugs that don't appear as such at the unit level. Two objects might individually be in perfectly valid states, but those states can be invalid at a system level.

Which is why there can be a false sense of security with unit testing. When you see 100% code coverage, it implies that you've tested every possible scenario, but with most non-trivial software, you haven't. Did you test what happens when the methods are passed in 100GB of data?

I've become a fan of using mutation testing to drive unit test coverage. The goal of mutation testing is to ensure there is test coverage around areas with a higher likelihood of being brittle. Work smarter, not harder.

> Once you've reached even 25% coverage, it's really dubious whether more coverage than that is actually telling you anything.

For sure there's diminishing returns on test coverage and I share your frustration with the myopic focus on coverage but there's no way it's anywhere close to 25%. Perhaps 25% is sufficient (or way more than enough) for new, small projects using languages with good compilers and type checkers. But it's no where near enough even under those ideal conditions when you have several generations of developers working on big multi-year projects.

Tests do reduce bugs. That's why automated testing was invented in the first place. Number of bugs vs company revenue isn't ignored because it'd make people look bad, but because it's an incredibly difficult metric to calculate.

I think this approach comes out of highly dynamic languages like python.

If a typo in a variable name doesn't cause any problems until it get evaluated, then you really need to execute every piece of the code to be sure that you don't have incredibly simple errors.

If you are using languages with a even a primitive static type system, you can catch a lot of errors at compile time (or when the interprartor loads them), which provides most of the benifits of 100% coverage.

> If a typo in a variable name doesn't cause any problems until it get evaluated

Linters even in languages like python will catch that kind of thing.

for undeclared variables PHP is even worse. It just keeps going and just gives you a notice!

PHP has a lot in common with BASIC at that.

IMO it's a huge problem that TDD and unit testing hey conflated way too often. TDD just means you write the test first and don't write code not required to get the tests to pass. It does NOT tell you what tests you should write. I for one am a huge believer in inside-out TDD. This would have you start with an acceptance test that uses the product as a user would. This may be exercise the feature or reproduce the bug. It's for you to decide what the right abstraction level is. I typically will end up testing edge cases either one level down in integration tests or unit tests, but it depends and there are no hard rules. It's all a trade-off. What's definitely wrong to me is retiring solely on unit tests for your TDD that's a disaster waiting to happen. TDD is about fast feedback loops and quality not about proxy metrics like coverage from unit tests. As others have pointed out, depending on your language you also might need more or fewer tests. When I write Ruby or plain JS, I'll write a lot, a lot of tests on different levels. When I'm writing Rust, I write very few and those that I do write will be focused on complex business logic (granted all Rust code I've written was for toy projects or puzzles where "business" logic might be the wrong term)

Edit: we've also corrupted what "unit" is referring to. AFAIK it originally referred to a unit of functionality. Many now take it to refer to a single class.

100% test coverage is really the starting point for testing. Once all the lines of code you've typed in actually run before they're delivered to the customer, then you can start adding the real test scenarios!

And how, pray tell, do you discover all of the scenarios that need to be covered?

If only there were some sort of coverage tool that could tell you...

You are forgetting one important part: unit tests serve as an always updated, always tested documentation of how you use the functionalities of your software.

I agree with your point on 3 - you should use the tests to start figuring out how your unit is going to be used. If you write code first, you frequently end up with untestable code that you need to mock the universe to get it running in artificial conditions. I don't recommend writing your tests first unless it's a trivial case, but you should at least have the unit calls in tests before you start coding.

This has been my interpretation of unit testing for some years now. They should be renamed to "Examples" or "Documentation Test" because all they really do is when I read them is (i) show how to use the unit (ii) and common error / success cases of that unit.

I would never really on them for actually testing the expected property of a public API for example.

Depends on how extensive the unit tests are. But, again, you also need to do integration tests for making sure you are not getting anything funny from external sources you rely on.

I like this post, because I avoided unit tests for so long lol. After finally doing them, I did notice that I would at least format my code better and this had other benefits. There would be cases I missed, made me start thinking about edge cases further in advance, as well as made it easier for other to read.

I think the primary point is made

> Or just write simple, single responsibility code so it can be tested easily

there it is.

This describes my experience with unit testing perfectly.

What I try to do is just write small functions that feel like they would be well suited for a unit test. I find that makes them much easier to read, and reason about. I don’t even bother writing tests.