Unit Testing is Overrated (2020)
tyrrrz.me
tyrrrz.me
1) Almost always better to be part of integration test
2) Offer no real ROI because the framework usually handles most of these issues gracefully to begin with, so its just a few hundred lines of cute code trying to mock out shit (or better yet getting multiple people in a zoom and wasting their time trying to test 4 lines of code - WHICH THE FKIN LIBRARY/FRAMEWORK ALREADY HANDLES)
3) Complete garbage. They write the code first and then add tests. That's fine if these were integration tests, but they run their little commands to print out a report that says 92% test coverage. Oh no No nO nO NO No meme-1.gif.jpeg. We have incidents regularly compared to my previous team, which was granted all Senior/Staff engineers and we didn't spend our time chasing dumbass metrics. If you want 100% unit test coverage, just do TDD and not this half-ass yolo cowboy approach. Or just write simple, single responsibility code so it can be tested easily. But nah, you know how it is.
These people are annoying as hell to work with, and unfortunately I work at a bigger company so this kind of BS is normal as people try to chase promotions instead of actually shipping quality products. I personally wouldn't hire people like this, nor keep them around this long.
If only there were some sort of coverage tool that could tell you...
Sure, I suppose one could say that any tests is better than no tests, but that also dismisses whether testing can be done better.
Code coverage is the worst. I've worked with codebases with various degrees of code coverage, and I've never seen a correlation between coverage and the number of bugs encountered. There may be an asymptotic nature to how much coverage is needed, but I would bet the threshold is low in a general sense. Once you've reached even 25% coverage, it's really dubious whether more coverage than that is actually telling you anything.
Of course, we could measure success by number of bugs and company revenue, but no, we can't do that; those could make people look bad.
Unit tests often skip the error cases that are really important for real life scenarios.
We've recently introduced a regression test suite for our data pipeline which simply compares all the data before with the data after running the pipeline. This gave us about 65% code coverage and the certainty that we're testing what actually matters for the product instead of how somebody decides to implement a function.
Which is why there can be a false sense of security with unit testing. When you see 100% code coverage, it implies that you've tested every possible scenario, but with most non-trivial software, you haven't. Did you test what happens when the methods are passed in 100GB of data?
If the junior dev took the time to understand basic DB performance instead of writing dozens of unit tests, then it wouldn't have happened. I blame the gurus who don't write code for living, but peddle magic solutions like "clean code", unit testing etc. to sell more books and seminars.
The extra time could then be used to learn basic Db performance, which is hugely important when tables have millions or billions of rows.
Stuff like indexes and isolation levels should be taught to all junior devs, it's just too risky not to.
For sure there's diminishing returns on test coverage and I share your frustration with the myopic focus on coverage but there's no way it's anywhere close to 25%. Perhaps 25% is sufficient (or way more than enough) for new, small projects using languages with good compilers and type checkers. But it's no where near enough even under those ideal conditions when you have several generations of developers working on big multi-year projects.
Tests do reduce bugs. That's why automated testing was invented in the first place. Number of bugs vs company revenue isn't ignored because it'd make people look bad, but because it's an incredibly difficult metric to calculate.
If a typo in a variable name doesn't cause any problems until it get evaluated, then you really need to execute every piece of the code to be sure that you don't have incredibly simple errors.
If you are using languages with a even a primitive static type system, you can catch a lot of errors at compile time (or when the interprartor loads them), which provides most of the benifits of 100% coverage.
Linters even in languages like python will catch that kind of thing.
a) identify bugs by failing on incorrect behavior, and
b) enable refactoring by continuing to pass when underlying code doesn't change behavior
Not that mocks aren't useful in some scenarios, but if you find yourself writing entire test suites based on them you're likely creating tests that are negative value. They won't catch undiscovered bugs, and they'll give you bad information when you're refactoring.
IMHO in my own projects I tend to write tests which test certain parts of a system as well as some dependencies. So usually I'd say they are integration tests instead of unit tests. But I can refactor stuff and once I break something it's usually detected by one of the tests. Maybe unit tests should test for certain cases border cases, which are hard to test with integration tests.
c) explicitly and accurately document expected behavior including in a variety of edge cases
...this is sort of implicit in your first 2 but I think worth breaking out.
Going for high test coverage (in places where I've done TDD-style dev) has really forced me to think through those edge cases and what I want my code to do in all of them.
I think negative-value testing comes from the mindset of "how do I write a test which exercises this branch?" but a positive version of that is "I put this branch to handle the case where foo is getting close to the boundary. Let's put in test cases with foo at and close to the boundary and make sure it does what it is supposed to." If the test focuses on behavior then you find the bugs and enable refactoring and you get high coverage because if you can't exercise all the paths probably some of them are dead and can just get deleted.
For 3) TDD has nothing to with test coverage or aiming for high coverage. The fact that TDD evangelists have tried to couple "good testing" with TDD is a huge scam. I don't care when you write your tests as long as they get written before the code is merged in. TDD is good for parts of the code which are very pure or where you know exactly what you are doing, but if what you are doing is more exploratory in nature or is not pure and has lots of external dependencies TDD is simply not practical for a lot of people.
Hopefully you also picked a language where a bigco has created tooling that lets you incrementally add static types -- TypeScript, Closure Compiler, Python typing and/or pytype, etc.
It taught me a couple of lessons:
1. Integration and System Tests are FAR more valuable than unit tests. By several orders of magnitude.
2. Performance Tests are vital. This is actually more true in a cloud world. There is only so much hardware you can throw at a problem before your costs are un-feasible.
I think the primary point is made
> Or just write simple, single responsibility code so it can be tested easily
there it is.
Edit: we've also corrupted what "unit" is referring to. AFAIK it originally referred to a unit of functionality. Many now take it to refer to a single class.
What I try to do is just write small functions that feel like they would be well suited for a unit test. I find that makes them much easier to read, and reason about. I don’t even bother writing tests.
I agree with your point on 3 - you should use the tests to start figuring out how your unit is going to be used. If you write code first, you frequently end up with untestable code that you need to mock the universe to get it running in artificial conditions. I don't recommend writing your tests first unless it's a trivial case, but you should at least have the unit calls in tests before you start coding.
I would never really on them for actually testing the expected property of a public API for example.
Integration tests need to be the default and people need to learn when to use one or the other.
I find integration tests paired with BDD do a good job of catching "code plain wrong" bugs as well as misused APIs and misunderstood specs. They're harder to build and run slower but the ROI is still higher.
You should know what result you want before you even start writing the code.
In theory, but IME about 50% of bugs end up being specification bugs at the high level anyway.
Once you drill down and start writing unit tests on lower levels to imitate higher level APIs that % only goes up.
If you first write the test and then try to satisfy them(test driven development) can work to an extent as it can divide the large task into small pieces where you get a prize each time but it also alienates the developer from the larger picture, diminishing their value to the project since they no longer can put their intellectual output into the project.
More green checks = more dopamine Higher quality code = long term satisfaction.
Code does things, tests verify that it does the things I thought it did. Getting a bit of validation makes me feel good while working.
If it now needs to do completely different things then it needs different tests.
Also, if you are making huge changes to tests whenever you change a single detail, that's an issue with your code, not testing in general.
Type checks are good too. Using a typed language is even better.
I'm not an all-TDD-all-the-time type by any means. You kind of have to at least have something to start writing tests against or it'll just be compiler errors all day long. But for the projects where I've most consistently found success in building correct systems, writing tests as soon as the basic interfaces are in place has resulted in more flexible, more reliable tests that last over the long term.
As for type checks...isn't that what a compiler is for?
That's literally the thesis of the Kent Beck book.
I'm not some idiot copy and pasting `expect(true).toEqual(true)`
I add functionality one test at a time. I get satisfaction of knowing it works as specified in the form of an increasing count of green checks.
I don't understand why it would alienate me.
Edit: I get that logically they are the same, but that's true of a half-empty/half-full glass. Psychologically they are different.
As you say, you don't care that there are 20k passing tests. You care just care that there is 1 failure.
If you don't believe this, just write some (good) unit tests, then make the change - it's very very likely they will catch some weird corner case you've forgotten even existed.
Yeah but the catch is that in order for your code to be "testable" you have to rewrite it completely in a way that comes with large sacrifices, making it much more confusing, abstract, larger and much harder to work with, in every way except testing it.
I totally agree that if you have to choose (and usually you do, unless you have unlimited time and funding), choose to implement end to end integration tests first.
It also helps form thinking about test cases and logic before writing code. Before I used exhaustive unit tests I would just start writing code right away and often times this would result in constant refactoring as the code evolved or during debugging. But unit tests encouraged me to scaffold out abstract functions and write tests before implementing the code. It got me thinking about what the logic and workflows then implementing the functions meant having the tests pass as verification.
Not only does it make your code more unit testable, it makes your code much more ad hoc testable. Again, I would not refactor a large, non-disposable application just to achieve this.
On the other hand I know developers that only unit test. Hundreds of lines of mock code, dozens of tests. No documentation, no sample configuration for the final application and dozens of data races because the unit tests don't cover multi threading. But hey an unusable buggy mess is a net positive as long as it makes the test coverage statistic happy and that is one of those things reported to management.
While a bit exaggerated one of the downsides of unit tests is that they quickly become part of a metric without regards for when they ad value.
They tend to have much higher value than pure unit tests.
I disagree with your assertion that integration tests offer higher value than unit tests in general. Don't judge all unit tests by mock code.
But, the failures from integration tests are often a lot harder to read. Even half-good unit tests tend to show exactly where the error is, and make the fix cycle much shorter. It is also much harder to write exhaustive integration tests for all of the inputs that might happen.
I, as a long-term qa-ish person (I usually write testing systems) see real value in having a mixture of both system, integration (multiple systems), and unit tests, as well as end-to-end manual tests. Anything big enough needs all of these.
Yeah unit testing as I see it is just optimising for the success of middle management, at the expense of both the organisation's goals, and the developers.
You are making predictability the only focus, because that's the metric which middle management is evaluated by. At the expense of performance, end product quality, and work satisfaction. "It was done in exactly the excruciatingly slow and mind numbing boring way, and was exactly as shitty as designed" = success for management.
If you would try to increase performance, end product usability, and/or work satisfaction, you would have to put the predictability at risk, which means that the success of the middle manager will not be the highest priority at all cost anymore.
The push for more testing and static typing (to degree) was always managers. They trade predictability for momentum.
The reality in both situations is that bugs happen. I've not noticed a difference in frequency of "critical" bugs in either scenario. The only difference is that at the no unit test/no e2e places, getting your code into production was trivial. Which always meant when bugs happen, they are fixed much faster than the unit test-everything place. We're talking minutes vs. days here. Because at a risk-adverse business (that is, any place more than a few hundred people), you have multiple stakeholders that have to sign off if you want to go take a piss in the bathroom.
I use strategy pattern with Java functional interfaces to avoid using magic mocking frameworks. That way, my mocks are just trivial impls of the plugin parts.
Sometimes you can't really avoid it. In those cases, I would judge how much of the effort is testing your code, and how much is testing your ability to write mocks. Unless it is a really important functionality, or something complex enough that it would be easy to get wrong, I wouldn't bother with most tests that need extensive mocking. Instead handle those later with integration / e2e tests.
I've seen code so mocked it was dubious that it could be testing anything, accurately at least.
Mocking should be kept to a minimum and ideally avoided.
Mocking is most useful when adding tests to legacy code, but new code should be designed so it can be tested without the need for mocking.
But it's very easy to overdo it and end up testing irrelevant implementation details instead.
Mocking outbound HTTP calls is worthwhile, but thankfully most testing frameworks I've used (at least in Python world) make that pretty easy.
No wonder they are not getting value out of unit tests.
However, there seems to be a general skepticism around property based tests in my experience. Any idea where it stems from?
But i'm not going to add an interface to every concrete class in my project, and design literally every component so I can mock it.
Writing software is a business, i can sprinkle TDD around for a high ROI. If I use it EVERYWHERE the ROI is very little if not negative.
But, VeryBigHugeBankCo requires me to have 70% unit-test coverage for 'new' code, as measured by Sonar or whatever, or else I am simply not able to deploy my change to PROD.
So here I am refactoring a DAO, using strategy pattern and functional interfaces, so that I can cover my DB interactions adequately for that 70% metric.
No, I can't just use H2 DB, as the SQL is Oracle-specific.
Yes, there is very little actual value in the new refactor, except to check the box.
This is how we do...
I have told mgmt that test code is equally likely to have bugs as production code, but, again, this is just box-checking, not actual attention to detail.
Why would using an in-memory database prevent use of triggers?
Here are qualities of good unit tests:
1. They run fast.
2. They help us localize problems.
The test that is too short is most likely not being thorough enough, or the functionality it tests is so simplistic the test likely brings no value.
https://www.artima.com/weblogs/viewpost.jsp?thread=126923
I think the properties you list are good ones, but if you drop the word unit from your post the advice is just as good and you can't get mired in ontological discussions
The design of those classes is bad, and the perceived need to reach for mocks (or interfaces) is telling you that.
The next step the author takes is to go for interfaces. That's not what I'd do. Instead, I'd decouple network from calculation with intermediate dumb data structures. The network side produces the data structure and the calculation side consumes it. The network side can further be broken down with a transformation of the API call result into the dumb data structure used for calculations.
This idea comes from an old Bob Martin presentation [1]. The whole thing is worth watching to put it into context.
This is the kind of thing that never seems to get discussed in these pieces praising integration tests over unit tests. For even better results, ditch the classes altogether and just use functions. It's quite surprising how badly designs come off the rails with the "everything is a class" model.
"Considering the factors mentioned above, we can reason that unit tests are only useful to verify pure business logic inside of a given function."
That just isn't true and it makes the rest of the blogpost also not true.
A unit test should test "a unit of functionality" not just a method or a class. Your unit tests also shouldn't be coupled to the implementation of your unit of functionality. If you are making classes or methods public because you want to unit test them, you're doing it wrong.
The exception is maybe those tests you are writing while you're doing the coding. But you don't have to keep them around as they are.
> it makes the rest of the blogpost also not true
what do you know of the rest of the blogpost if you stopped reading?
HOWEVER, I have skimmed through it by now and the last paragraph is actually quite good. It takes a while to get there and the chosen examples aren't great, though.
> That just isn't true and it makes the rest of the blogpost also not true.
It's not? Writing unit test is all about minimizing side effects and interaction with other systems.
Not really. Writing unit tests is all about verifying behavior from combinations of inputs, and side effects are inputs as well.
You can write unit tests that inject delays and timeouts and retries and throw exceptions under specific circumstances.
Well, except it's not. If you use a side effect, you have to account for it in some way via mocks or whatever.
Me too. The author shows a deep misunderstanding and even obliviousness to testing, particularly from a practical point of view, that takes any credibility from any argument listed in the article.
It's even perplexing how the best example the author could come up with was this absurd chain of strawmen that a) it's impossible to test code that uses instances of HttpClient, b) dependencies used in dependency injection "serve no practical purpose other than making unit testing possible."
There are plenty of people talking about unit tests. This article is not one of which justifies any click.
Main reasons to have dependency injection is to have properly runtime switchable code with less shared state. That it makes testing easier is a side effect.
I've heard this a lot, but how do I test private methods right? If my public method just calls 8 private methods (which each call out to a bunch of other private methods), how to I test to make sure all those private methods do what I want them to do so that I know which one is broken when my public method breaks.
There was recently a submission here on HN that talked about this and different perspectives on that topic. It's not a universally shared opinion, surprisingly.
Then there's the world of behavior / property / invariant -based testing which slims down your tests to essentially data generation and testing observable behavior which seems like magic to people still.
You don’t. You test only the public API. This way, you can refactor the public method to your heart’s content without any tests breaking.
The reality is that these private functions are building blocks that can be easier to test. If you only test the public methods, testing gets a lot more challenging and the methods are more difficult to test and the tests classes get a lot of more complicated. You end up creating more mocks or other doubles and bending backwards.
So how do you test private methods? You don't, with the exception of testing during development, but those aren supposed to be kept around.
Others have pointed out that you test only public methods. The thing I'd add to that: Having large classes is a code smell. If your class is big, it is probably several classes in one. Break it up accordingly. Some of your private methods in the big class will become public methods in the small class.
What you describe is white-box testing, where the test is coupled to implementation details of a class. This gives you more fine-grained reporting in case of an error but it makes it impossible to refactor the code safely. So I don't think that is a worthy trade-off.
If you often have hard-to-locate bugs in particular large component, the solution is probably to refactor into smaller more loosely-coupled components with well-defined interfaces.
Scala is great about making these functions generic. Java has an issue with this where lots of things are easy to get burried but hard to reason about pulling them out.
This is frankly the easiest and best bang for buck.
Unit tests force you to make your code more flexible, they're fast to write, fast to run. Maintenance depends on how overboard you go with your verifications - try to verify the most important parts.
End to end tests require no elaborate scaffolding of internals and allow you to test it as a real user interacts with it. Generally fast to make, slow to run, and maintenance depends on how disciplined you are with stable identifiers/interaction points.
As for in-code integration tests - I love the idea of them, but they're absolutely miserable 9 times out of 10 due to extremely convoluted processes to "bring up" parts of your application. If you use DI it shouldn't be as bad, but almost nobody does and it becomes a total clusterfuck not worth the maintenance burden.
As an idealist, I want all 3. But most codebases frankly cannot support all 3, so the most value will come from whatever is easiest. I've found that's most often unit and e2e.
"Aim for the highest level of integration while maintaining reasonable speed and cost"
My experience mirrors the author's. In any "real" business application, the unit tests end up mocking so many dependencies that changes become a chore, in many cases causing colleagues to skip certain obvious refactors because the thought of updating 300 unit tests is out of the question. I've found much better success testing at the integration level. And to be clear this means writing a tests inside the same project that run against a database. They should run as part of your build, both locally and in CI. The holy grail is probably writing all your business logic inside pure functions, and then unit testing those, while integration testing the outer layers for happy and error paths. But good luck trying to get your coworkers to think in pure functions.
Good! They shouldn't do the refactor.
Because "obvious" refactors often introduce bugs (e.g. copy/paste errors), and if developers can't be bothered to write tests to catch them, they're going to screw over the other team members and users who will be forced to deal with their bugs in production.
> The holy grail is probably writing all your business logic inside pure functions, and then unit testing those, while integration testing the outer layers for happy and error paths.
So settle for half a loaf.
Write all the easy unit tests first. The coverage will be very incomplete, but something is better than nothing.
Write all the easy integration tests next.
Never write the hard tests if you can help it.
> Because "obvious" refactors often introduce bugs (e.g. copy/paste errors), and if developers can't be bothered to write tests to catch them, they're going to screw over the other team members and users who will be forced to deal with their bugs in production.
In my opinion, useful tests should be able to survive a refactor. That is the only sane way I've ever done refactoring.
If I'm doing a large refactor on a project and there are no tests, or if the tests will not pass after the refactor, the first thing I do is write tests at a level that will pass both before and after refactoring.
Rewriting tests during refactoring doesn't protect from regression on my experience.
I would not rule out a refactor merely because I'd have to refactor some unit tests too. That's just part of the cost benefit analysis.
> Rewriting tests during refactoring doesn't protect from regression on my experience.
Your experience is completely at odds with mine. Every time I change code, there is the possibility for simple errors such as copy/paste mistakes. Trivial, cheap-to-write unit tests have saved me time and again from having to debug something down the line.
Overconfident devs who act as though they're above making such simple mistakes make for bad team members.
Tests that will survive a refactor are the most important tests to have.
The other tests are, at best, a false sense of security and often an active detriment that slows down future development. They might sometimes catch actual mistakes, but just as often they fail when nothing is broken, leading to the tests not being trusted and broken tests being updated even when something was actually broken.
*sigh*
I know where you're coming from. This is the classic argument for limiting unit tests to black-box testing of public APIs exclusively, avoiding clear-box testing altogether.
I agree that it's possible to write absolutely wretched fragile clear-box tests. And I agree that if you have a black-box test and a clear-box test which provide equal validation of functionality, the black-box test is superior because it will survive a refactor.
I generally dislike absolutist rules of any kind when it comes to unit testing and prefer to think of things in terms of ROI. Sometimes you can add a lot of value with a clear-box test because the functionality is impossible to write a black-box test for without a ton of extra work and time.
But sometimes you may be in an environment where absolutist rules are the only way to go.
& if your tests arent catching those bugs and require extra maintenance to go green again you are doing them wrong.
I avoid tests (aside from hands-on end user testing) as much as possible, actually, since they rarely seem to tell you anything you'd didn't already know.
I think this is just one of those cases where there is a context-sensitive strategy to testing. It depends completely on the cleanliness of your code and experience working with it.
I've come to a similar conclusion. Functions don't necessarily have to be pure in the academical sense, though - but I feel like the more the business logic is decoupled from dependency injection and the less it is relying on some framework, the better.
It makes testing a lot easier, but also code reuse. I've just been writing a one-off migration script where I could simply plug in parts of the core business logic. It would have been very annoying if that was relying on Angular, NestJS or whatever.
Then you get to use fuzzer & Arbitrary for basically what is a coverage guided property based test.
But it's hard to maintain that idea at all times when you are writing.
Unit tests are very cheap, and people love cheap things. The problem is that they also provide almost no value, so their cost and benefit are on similar levels and they can easily get into negative net value.
I've definitely worked on projects like this (actually, I'm working on one now at $dayjob), but I found that with a little bit of effort you can get this right, and usually isn't that much effort.
What I often see people do is "oh I need a thingy for this test, and a thaty for this other test" and create it ad-hoc when needed, instead of creating a good convenient API to do all of this, which is then also used in the application itself.
The big advantage of these tests is that they tend to be a lot faster than E2E tests, at least in my experience, and often also easier to reason about once you get it right.
1. Unit tests for code with no side effects (code that takes well defined input and produces well-defined output) are easy to write, easy to understand, and I've never regretted writing them.
2. I've had good experiences transforming as much error-prone logic as possible into code with no side effects.
3. Integration tests that execute in an environment that is similar to production have proven to be incredibly valuable, especially when refactoring. When possible, I prefer to make technology-stack choices that make such an environment quick to setup and teardown such that these tests can run in seconds.
4. Many bugs occur because team A makes bad assumptions about how team B's system behaves, and they encode these assumptions into their (passing) test cases.
5. Unit tests with lots of mocks should be avoided; their cost-to-benefit ratio is terrible. Sometimes the best solution is to delete these tests. Relying on mocks for a few error-case tests that are hard to reproduce on the real system is ok.
6. If they cannot be avoided, fake implementations should be written by the same team that writes the real implementation. They understand how it actually behaves much better than their users, and are in a good place to reuse critical logic from the real implementation in the fake to make it more realistic.
7. If my project includes firmware running on custom hardware, building an emulator for the hardware that can run on standard computing infrastructure is valuable for writing test cases against.
8. More tests or coverage is not always better. We have a finite amount of time to improve our systems, and it is our duty as engineers to ensure that the benefit we get from adding a specific test-case is higher than using that time to make some other improvement to the project.
Tighten up the development loop (manual e2e is often too slow)
Prove to yourself that the code does what you want
Communicate intent to other programmers
Documentation to other programmers
Help programmers debug regressions
I really think this holds up because, really, who except for programmers that can read and comprehend unit tests are going to trust they verify a product works? The more distant a stakeholder is from a project the more holistic the testing needs to be in order for that stakeholder to trust it.
UT is not the panacea and won't do your laundry for you, but well-tested code is better than sloppily- or un-tested code. Always will be, IMHO. UT is the core of that, for languages where a unit of functionality can be executed directly. Integration tests are also critical, but IMHO it's foolish to do one or the other.
"But dependencies ..." yeah that's been solved many times over with mock tools and such, and refactoring to reduce complexity. If your code is too complex to test properly, it's too complex to release. Don't be lazy, do it right.
I might not be too lazy - but I have to collaborate with 5-10 people.
Well-tested code is also some kind of utopia, no one have seen such code but everyone is claiming that once you get there it is all unicorns and roses.
I can write perfect code if I do it alone for my toy project - when I have team of people that has to agree on stuff it is not going to happen.
I think this title could maybe be rewritten as "Unit Testing that Requires Mocks is Overrated".
Unit testing something that has no (or very simple) dependencies is great. For example: - some kind of encryption that takes in a string as a key - serialization that expects the same output as the input passed in - a transform function that takes in one type and expects a different type out
As soon as you rely on a DB, file system, etc... you're probably better off with an e2e test.
At the end of the day, it comes down to data contracts. This could be the functions a package exposes or the GQL/REST/gRPC/whatever API. That is the most important thing to not break the behavior of. Write good tests that target the external facing data contracts and treat implementation like a black box and you'll be empowered to do the important things that tests should enable like refactoring, reworking abstractions that may have made sense at one point, but no longer do, and let your codebase evolve as you learn.
Tests that are a barrier to reworking internal abstractions are not good tests.
All of this to achieve a very arbitrary 80% coverage that's required by business on a few REST APIs that's not even ours!! And don't get me wrong I get the importance of testing but the enphasis on unit testing these days seems ridiculous
What kind of software do you make?
Where I am, I was still very much learning after 6 months and it wasn't until a year or so before I became really effective.
I don't think the concept of "unit testing" is the main problem here.
Unit testing is not overrated, if you feel this way you likely just suck at optimising your suite for high RoI tests or chase some stupid metric like code coverage getting mad when you find out you wasted a bunch of time writing worthless tests.
Figuring out where your bug is is so much harder when you have to do it though the lens of other code, whereas in your unit test, you can just see "oh, I'm returning foo.X, not foo.Y".
It also makes sure you actually can construct your object at all without dragging in a dependency on the entire system. Code without tests tends to accrete things that can only be set up in a very long and complex process. This is both hard to reason about (because your system can only be thought about as an ensemble) and fragile when the system changes: you now have to unpick the dependencies to allow your "SendEmail" function to not somehow depend on an interface to some SPI hardware!
But there's certainly value in not spending hours testing obvious code to death: a getter is almost certainly fine, and even if it isn't, the first "real" test that uses it will fail if you're returning the wrong thing. But if, down the line, you do find a bug in it, then something was probably not that obvious!
It's also valuable to have some testing framework available, just so that it's then easy to write tests when they're needed (which comes back around to "make sure that your objects can ever be constructed" thing). Not all unit tests have to be preemptive, even if the presence of the ability to quickly write them is.
In Unit testing, you should only be testing the code you wrote. (Contract testing is more of an assurance test) In Java you have to write unit tests for your getters and setters because they might do more than what you expected (as that's within the realm of possiblity) If you find yourself writting getters or setter tests .. you should be looking into a better language like Groovy, Scala that do it for you, or something like Lombok or Records in later JDKs.
Yes, what you need to do in the end, is to use your judgement. But you will get a surprising amount of pushback against this obvious common sense solution. From management who thinks it's too risky, and from developers who have chosen a technical career just because they want to rely on their intelligence, and black and white thinking, and now you're asking them to make trade offs and decisions which is management territory. The way out in my opinion is to make technical leaders that have actual authority, down to the details.
Unit tests help you to design your classes based on use cases—if a simple class is hard to test you should think about whether it might have a simpler design. If you have a brute forced design and each of your classes have dozens of dependencies and your tests require pages of setup and are hard to reason about and are tightly coupled to the implementation then the tests might not be providing any benefit at all.
For backend testing, I've had amazing success following a few simple principles: 1. Abstract and mock code you don't control that require network calls. 2. Use frameworks that support database abstraction out of the box 3. Test end to end functionality. Every test should basically be getting your DB into a specific state, then hitting an API endpoint and validating the output and/or state change.
It's not that hard, you can get decent and worthwhile code coverage for not much work, and even if you only write a few tests with low coverage, you can easily add regression tests later if you notice a bug.
If you encourage developers to write tests with the mindset that "these should make my life easier", you'll get better and more maintainable tests.
If you encourage developers to write tests "just because" you will get very useless tests.
- There's no consensus on what a "unit" is.
- There's no consensus on how much coverage is needed.
- There's no consensus on what should be mocked.
- There's no consensus on if private methods should be tested independently, even though they obviously shouldn't.
- It violates "don't repeat yourself".
- The developers who write the tests usually write the code. Most bugs occur when developers overlook a use case. Do the math.
In fact the whole thing then becomes a standalone function which takes Location and other relevant parameters and returns the computer sunrise/sunset values. Pure function and super easy to unit test.
If one needs to do a lot of "mockups" for your unit tests then maybe one needs to consider the API and class design. Removing needless coupling helps testability by removing the need to use mocks in the first place.
A role of one unit inside this component is to use the provided coordinates and only perform the calculation.
Another situation - where my wife used to work in medical devices. Feature development velocity was a problem because there was a queue on the very limited set of devices (CT scanners, PET scanners, etc.) that the team could use for testing. Debugging was very hard. Fixing bugs was hard because to debug you needed a device. With unit testing and mocks they made a lot of contention on the devices go away.
Write your unit tests. It will help you and the people coming in after you to maintain the code.
Unit Testing Is Overrated - https://news.ycombinator.com/item?id=23778878 - July 2020 (387 comments)
Maybe that is the standard understanding, but I don't think it was the original meaning, and I don't think it is a helpful definition. Testing only "smallest separable parts" is what leads to an over-reliance on mocking and brittle tests.
Unit test should test a "unit of behavior", not a unit of code. That is, a unit test should test a single assertion about the behavior of a component. The test should not care how the behavior is implemented, i.e. it should be possible to refactor or fully rewrite the implementation without changing the test, which means the test should not care if one or many sub-component are involved (as long as no external systems are involved).
For languages like typescript and kotlin, I’ve found them useful in situations where I don’t have a repl available.
Custom react hooks are much easier (and faster) to develop in unit tests without having to load in a whole application. Exploring data models and their functions before implementing views is easier in unit tests.
I find unit tests to be a really productive playground and I think it’s great to have a place to later jump in and extend functionality in a safe space.
I also think of tests as "scaffolding" when building a skyscraper. They make it easier to access certain "things" before they're fully complete and are a cost that makes building the actually building less risky and quicker. Why does no-one question the value of paying for putting up and tearing down a scaffold when building or repairing a building, but can't see the value of unit testing?
It's certainly not compatible with what Michael Feathers describes as unit tests in his "Working Effectively with Legacy Code" classic, a book I consider the bible on unit tests (even if a bit dated by now).
If anything, many of the things the article describes as drawbacks of "unit tests" are exactly the things that Michael Feathers seems to describe Unit Tests are "not" if they are to be effective as unit tests.
Obviously I'm being awfully vague here, but, yeah. That was the feeling I got from the article. It's just another case of "My particular definition of X, which is not necessarily what others understand by X, is bad. Use what I define as Y instead, which may be what some people used to mean by X in the first place, but let's call it Y from now on because X is bad."
- complex pure functions. For instance, I have a street address "cleaner". The input is an unclean street address (weird capitalisation and so forth) and ouputs a clean address. Unit tests are perfect here because they're fast, and it's an isolated piece of logic that has no side-effects (no writing to DB or network request).
- wrappers around third-party libraries. Yes, in theory you're supposed to trust that a library will respect its API unless major version changes etc. In practice, it's not the case. And I'm also lazy and don't want to read all the release notes of all the libraries when doing updates. By wrapping a utility function from a third-party library and unit testing that function, I can have real confidence that the library still works in the way I'm expecting it to work. And it fails loudly when it no longer does.
Outside of these, I only have end-to-end tests that test the whole stack: send a GraphQL query and check that the resulting JSON is what I expect.
That's an "acceptance test" or "fit for purpose" test in case you're looking for industry terms.
Closely related is "law of demeter" (the law of one dot), as it buffers you from changes in dependent API's.
OP criticizes dependency injection, and comment that its unique purpose is making UT easier. I strongly disagree. I've been able to easily replace web frameworks and OORDB libraries using DI, and even develop business logic before selecting infrastructure technology. In that sense, the fact that UT are easier to write is an indicator that the system is easily adaptable.
A good UT allows me to reduce integration testing to the minimum. What my IT do is verify that two systems can exchange messages, but as soon as I get a request or response representation stripped from any I/O reference, it's UT time for me.
Unit testing massively reduces all of these. Therefore I assume people that don't like unit tests either enjoy doing these things or are masochistic.
https://tyrrrz.me/static/b606743d039cfb56391c5793d0dda8f2/bb...
It shows test cost vs test isolation. I agree with the assumed correlation. My experience though is the real correlation is above the assumed correlation, not below it. My guess is that the author's organization moves part of the cost somewhere else where the author doesn't have to see it. At the start of my career, I was one of those people who had this cost dumped on them so that the devs could ignore it: I was an on-team QA engineer.
This software is still buggy and broken.
Ergo, unit testing is not working.
Testing has nothing to do with engineer ability or experience. Regardless of that you can always make a mistake. And even if you are a rockstar Sr dev who never makes a mistake, in the real world any piece of code is gonna get worked on by people of all skill levels and experience so the idea that "a senior engineer wrote it so we can just trust it" doesn't work even if one accepts the assumption that you can just trust the code from an experienced dev. I am against 100% coverage requirements as well (except for perhaps dynamically typed languages) but I would still expect a reasonable amount of coverage and that coverage include good tests.
They are in certain agile circles. Ones I used to work in.
Absolutely agree, and when we have identified trustworthy software engineers, we also need to give them authority. This doesn't work however, because it puts the middle manager on the spot to make this judgement call, and it also goes against the grand master plan of software engineering which is to turn it into outsourceable factory work.
I'm working on an app that has 200+ tables and requires a lot of data to be in the database before anything even works at all. Of course good tests interacting with the database need a fresh database every time so even a test suite of about 10 tests can take 10-20 minutes already.
Slower, but the test suite is far more effective at catching bugs.
These tests do take more up front engineering but since they arent pale imitations of the code they also dont need to be totally rewritten every time the code is refactored. Higher capex, lower opex.
No, all you have to do is spin up your service under test, its database and the service bus/event hub/whatever you use to communicate between services.
Nowadays with Docker it's really easy and fast to spin up test instances during your test run. (4-5 LoC with a good library in C#).
Since 1 or 2 years I'm doing the exactly same thing on my projects and it works great to speed-up development process and if a bug pops in production it's really easy to add it as a future test case.
You're only attached to the input request (endpoint, query params, request body) and the response body. But that's ok because it's already a part of the contract between the API and its consumers. That's it.
This is what writing useful tests mean, from a client perspective I just want to understand what's valid and what's invalid.
Also useful to make sure we fixed specific scenario bugs.
I've seen a lot of people doing things wrong but we can summarise in two cases: - "We've got 100% coverage, it must mean we've got no bugs, right?" - We've got tests covering happy path, everything else is tested manually. It takes forever and it's not shared anywhere, but "trust me" I've tested this.
Tests themselves are not overrated, just like with everything else in engineering one has to use some common sense when choosing what to test, and what not, it's as simple as that - but because this can't be easily bottled and prescribed as a rule of thumb, managers/leads prefer to cover their asses by insisting on 100% coverage...
However you need integration tests and end-to-end tests to improve software correctness.
That's actually almost never true - abstracting the sorts of external dependencies (like databases and web services) that actually cause problems for unit tests make the code more manageable in terms of maintenance and troubleshooting.
The problem is when people go mock-crazy and mock every class that the unit depends on. Rather, just mock the things that can actually make the unit test fail if they're unreachable for some reason independent of the actual code.
That suggestion was met with not only resistance, but almost hatred. I think some of them got way too attached to these tests and could no longer see the forest for the tress.
https://quii.gitbook.io/learn-go-with-tests/meta/why
The book is Go specific but there's a lot of wisdom in this particular chapter that applies to any language.
Just consider that in many cases code lifespan is 20% dev , 80% maintenance.
My experience is the exact opposite. Actually my experiences are the opposite of many things in this article.
instead of having separate test/ directory with multiple unit test files, I put unit test code right in the source code(at the end of the file), so when I modify the source I can easily update the unit test in the same place, and they never get out of sync. It worked for me very well. They can be turned on/off easily too. Sometimes it's easier to understand the code by reading the unit tests than reading the whole lengthy code, and it's helpful when they're in the same place. For me, unit test code is part of the real code, not something 'additional'.
https://littlegreenviper.com/miscellany/testing-harness-vs-u...
Consider trying to write tests in a non-trivial Spring Boot app without mocks
You will here the exact opposite from proponents of test-driven design.
The most complicated code I've seen is integration-test heavy code. It turns into a big ball of mud very quickly as new features added simply add more lines of code to already long procedures. Over a hundred lines of database calls, HTTP calls, variable bindings, conditions and branches. Code like this doesn't compose well and is a refactoring nightmare.
Where writing testable code can lead to complicated designs is where developers are not thinking about abstractions and are, instead, introducing indirection. This is a common solution in many object-oriented languages with inheritance and virtual methods. There's a whole essay and reams of textbooks on what makes a good abstraction in a precise way but most developers think abstraction means indirection.
However, if developers are concerned about writing abstractions it often reduces indirection and leads to easier-to-test code. This is because abstractions introduce precise semantic layers and code under test then shouldn't be concerning itself with underlying layers. A test-driven approach can lead to better designs but I think the claim that it will guarantee them is a bit over-sold.
> Unit tests are expensive
It appears the author means in the form of developer time. In terms of execution time if the tests are slow there's an underlying problem that should be addressed.
In general is it more expensive to ship errors to production or to take the extra time to validate your assumptions, invariant properties, etc?
> Unit tests rely on implementation details
If you write them that way, they will. If you spend most of your effort mocking out dependencies in your code it means you're missing opportunities for abstraction. Users of your code must care how the underlying dependency is implemented and there fore there is no abstraction.
If folks are forging ahead with mocking out dependencies like this across the code base you have bad abstractions and tight coupling in your code. You could step back from this and treat it like a bug and fix it... but this isn't a problem of the practice itself if people are ignoring the signs.
The authors solution to their straw man is... more end-to-end, behaviour driven tests. While valuable these are demonstrably more time consuming activities: both in execution time and in developer time.
I think a good peppering of unit tests is healthy; it aids in refactoring different layers of the application and gives decent indicators if your refactoring changes behaviours in a rapid-feedback loop.
However I think unit tests are... still not great. There's not a lot of empirical evidence to support them in reducing error rates in code. And I think that tracks with their use in verifying systems: at best they're an example of a single input producing a single expected output... but the domain and range of most units is quite large and even a hard-core unit-testing team will not cover most of the problem space. In essence they only prove the existence of errors, not their absence.
For that we need better tools like property-based tests and verification techniques like refinement typing, separation logic, etc as suits the problem space.
TL;DR The guy doesn't understand how to test code and he doesn't understand test isolation. He responds to requests to eat his vegtiables as "but I dont hafta"
Unit tests are intended to test small mostly uncoupled parts of the code. (Usually scoped to single functions). Integration tests confirms the combination of functions, or the class in a context.
Testing of user behavior is intended to be done in the functional tests, and only that. When you get to functional tests you should assume that most of the code is trustable underneath, as that the tests support that.
The intention of the separation of the testing strategies means you don't have to do a ton of combination tests in a higher level (where the test is more expensive to run, write and maintain).