Unit Tests Are Overrated: Rethinking Testing Strategies
thenewstack.io
thenewstack.io
I tried finding non-platitude insights but couldn't.
His conclusion is: > By embracing a more holistic testing approach, teams can deliver better software faster. As with so much in life, the key is using the right tool for the job, not just the tool that is most convenient at the moment. When it comes to testing, smart and strategic beats brute force every time.
Note how this statement conveys zero information and could be said about literally anything.
https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste...
One retort might be that sometimes code defends against malicious actors who, say, have direct access to private endpoints etc., but if you're defending against such threats, either that requirement drives your functional spec (e.g. banking infrastructure) or maybe you're taking things a little too far.
Sad truth? Only? How about an alternative interpretation: 90% of teams don’t need or want to release at least daily. And so they don’t.
Continuous delivery is something that 90% of software teams don’t need and shouldn’t waste time on. It’s been overhyped.
You don’t need to release to users, that can happen later.
Every team I’ve been on that could release daily never wanted to go back to anything less.
The tooling is still too hard for mass adoption by teams, but the benefits are real.
Yes, testing, especially unit and integration tests, are a sine-qua-non for quality work. But so are automatic proofs of absence of certain errors (vulgo type checks) and non-functional tests.
But all this has been at least partially resolved. What we are still sorely lacking is simulation. Everyone who ever worked with a staging environment knows how limited this usually is. What I would like to work with before I retire is a tool that lets me standup a complete copy of the production environment, isolated from the real world of course, with all external services automatically mocked. Now that would help to improve the quality.
It is mostly the human that is most expensive to replace with a simulator, apart from simple end-to-end or stress tests.
Is anyone working on a reinforcement learning agent for proper monkey testing? :)
I honestly can't imagine any software product with test automation that would require more than a real day to pass, never mind a quarter. Not unless some person or bureaucratic process was deliberately slowing it down.
>But all this has been at least partially resolved. What we are still sorely lacking is simulation. Everyone who ever worked with a staging environment knows how limited this usually is. What I would like to work with before I retire is a tool that lets me standup a complete copy of the production environment, isolated from the real world of course, with all external services automatically mocked. Now that would help to improve the quality.
There are plenty of tools that do this. I run simulators to catch sent mails, pretend to be browsers, mock REST APIs, etc. This is the bread and butter of test automation. The actual scenario writing I find to be the easiest part.
The real problem is that this stuff is usually done really badly by cheap, poorly skilled people who will never earn as much as a dev and who almost never do this part of the job properly.
The actual solution is in my username.
The weird thing is that this really isnt a technical problem. It's an issue with social class.
IME almost no APIs actually provide that level of sandbox support so that immediately means you can't run comprehensive, fully automated tests of those integrations.
Of course you can mock those endpoints instead. But then you're not really testing the integration at all and conveniently ignoring the place where most of the real risk is often found.
It didn't work exactly right of course because the docs weren't exactly right. Ultimately all we could do was try to fit to the docs, manage expectations, release into production, see the errors, log as much as we could, try to cope with the fallout and update the mocks accordingly afterwards. For each problem/error we ran into, this took under a day.
A longer test/release cycle would have made this problem far worse not better.
Indeed. That approach also won't pick up changes to the remote API that break your integration.
An alternative strategy in this situation if you have access to some kind of remote sandbox - but maybe just a single stateful one that must be shared by your whole org - is to factor out your integration with that service into its own component/service and deploy controlled updates to that service less frequently and after running through a comprehensive integration test using the real sandbox.
This probably involves a much lower release frequency but it also keeps your integration as stable as possible for your internal clients and tested against the real external service. If you're talking about remote services that do things you can't trivially revert - like physically transporting some object or causing money to change hands - then the strategy you described of trying to manage expectations while testing in production is probably not an acceptable alternative anyway.
Common but not necessarily best.
Let me pose a question: if refactoring is "changing code while tests stay green", then how do you refactor when any change to public methods always breaks tests? When "extract class" means a change to test class structure as well?
Testing at this close-coupled level far from mandatory. IMHO, the use of the word "unit" is deliberately vague: if it was always a "class test" or "method test" then it would be called "method test" and "class test" not "unit".
"unit" means "you choose". it means "a chunk of useful functionality", in problem domain or business terms. It means "can I test this in isolation from e.g. file system, http services and other concurrent tests", as per Michael Feathers's guidelines (1)
I'm not suggesting that you should never unit test at a close level of class and method; I am suggesting that you might benefit from a majority of unit tests being not at this close level.
1) https://www.artima.com/weblogs/viewpost.jsp?thread=126923
Same for unit tests and any other practical matters that have turned into religion.
Plus there's the little matter that you tend to think that your application is tested if it has automated tests. Then your colleague in sales who clicks like he's having an epileptic episode comes over and your app crashes 3 different times [1].
[1] True story. I used to ask him to click 2 min on whatever I was working on when he passed by my desk.
[2] PalmOS had the gremlins tool. To my knowledge no other platform has it.
If you want to make sense of "Agile" read "The Agile manifesto" (1), it's not long even with the 2nd page (2).
but yes, the modern criticisms of modern "agile" are pretty much the same as the problems that caused that agile movement in the first place. And are starkly at odds with the Agile Manifesto, e.g. "Individuals and interactions over processes and tools" and "Responding to change over following a plan" vs "Our tool, Jira, enables micromanagement of human resources over a long-term plan"
Yeah, too bad modern Agile doesn't have much to do with the original manifesto...
Hit multiple keys on the keyborard at once and do that for a few seconds.
Then if it p[assed that he would get down to more structured testing.
Testers have to assume the user is an idiot and also will always do the wrong thing.
But that value is not in yet again, before each release, taking a day to run through an excel spreadsheet of checks that should have been long ago automated into a 2-minute run.
But some tests will take time to run so a day turn round is a quick turnround.
If you have databases involved it will not be a 2 minute run - the network latency will make the queries take time. Note these need to be run as integration tests so mocking is not sufficient.
This becomes more difficult to reason about when what you build is not just a library but something more like a web service or a GUI/CLI app. For a library, a unit test may be able to cover public API and be an end-to-end test in effect, whereas for a more complex product intended to exist over time in production unit tests do in fact provide less of a guarantee, and versioning public API becomes difficult (unless you immediately drop support for any non-current API version).
Contract changes is one reason for such a code change, but there are many others - including bug fixes, error handling, handling input cases that were not well supported before, performance or readability improvement, changing to use a new version of an internal library with a new contract, and pure refactoring to e.g. extract common code, with no other logical change.
it is useful to do such incremental changes supported by tests on a "publicly exposed interface" that does not change frequently, instead of tests coupled to classes that are not not part of a "publicly exposed interface".
The biggest sin of the whole IT sector at the moment is lack of quality and this thinking drives it further towards hell.
Do less, do it slower but do a better job please.
A good example is security. Sometimes it's righ to ship stuff with security issues that you know exist and might even be 10+ million dollar issues if discovered quickly if for example fixing thoses issues might interfere with adoption because they make user onboarding slightly more annoying.
For example, better use of types to make sure applications physically can't get into a bad state will always have better returns in terms of velocity and correctness than checking after the fact with unit-tests. It has the additional benefit of being compositional, unlike unit tests.
This can be done for data representation but also transitions of state, which is all of software engineering outside of IO.
They also overlook a large part of the value of unit tests, which is fast feedback during development time. We could push all of our tests up to the browser level, but then I lose the almost instant feedback that unit tests give me.
Sure, if your tests mock everything, and only test individual methods, then they probably are brittle and provide little value over more 'expensive' (e.g. slower) tests higher up the test pyramid.
If the tests only interact with the public APIs (e.g. the 'ports' in a ports and adapters architecture), then they are less coupled to implementation, can represent real business scenarios (instead of 'input x gives result y'), and add more value.
You've fallen into the very trap of terminology the author mentions at the beginning of the article.
Testing groups of functions or classes together is still a unit test, by Wikipedia's definition at least
"unit testing is a software testing method by which individual units of source code—sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures—are tested to determine whether they are fit for use" [1]
and by Michael Feather's definition
"A test is not a unit test if:
- It talks to the database
- It communicates across the network
- It touches the file system
- It can't run at the same time as any of your other unit tests
- You have to do special things to your environment (such as editing config files) to run it." [2]
Indeed, it's the inevitable result of doing TDD; the refactor step is likely to break out smaller classes/functions already covered by the existing tests. If we write new, more granular tests every time we do the refactor step, then we end up with brittle tests.
[1] https://en.wikipedia.org/wiki/Unit_testing
[2] https://www.artima.com/weblogs/viewpost.jsp?thread=126923
Edit: To clarify, I'm not talking about calling the public API over the network, I'm talking about the public methods that consumers (including your network layer) would call.
That might be where the confusion lies here.
That's not what it means, and the author clearly indicated it didn't?
When your colleagues are talking to you, you're going to be constantly misunderstanding them if you just make up your own definitions, even after they explained them!
When I was 14 I believed @ meant about and now I no longer use @ to mean about.
It is perfectly fine to test multiple classes as a unit, in fact you probably already do so, as I would be extremely surprised if you mock your string class in all your unit tests.
https://martinfowler.com/bliki/UnitTest.html
Martin Fowler
Although I start with the notion of the unit being a class, I often take a bunch of closely related classes and treat them as a single unit.
https://medium.com/@_ericelliott/i-use-the-well-known-defini...
Kent Beck
Unit tests test individual units (modules, functions, classes) in isolation from the rest of the program
Such great evidence.
Then, he goes on to, and let me emphasize this, defend against the criticism that other people make that this type of test is not a unit test. (sound familiar?):
Indeed using sociable unit tests was one of the reasons we were criticized for our use of the term "unit testing". I think that the term "unit testing" is appropriate because these tests are tests of the behavior of a single unit. We write the tests assuming everything other than that unit is working correctly.
I'm not sure how thoroughly you're looking to be refuted here but I feel I can't quite do a better job.
Funny thing is, this is not even a very deep insight: the answer to nearly any question in software design can be boiled down to: it depends. This discussion is just the unit-test rendition of "it depends". Why are you so hell bent on having it exactly one way?
That's the worst idea ever. Testing requires completely distinct set of skills different than development. There are 10x programmers but there are also 10x testers.
Developers must be accountable for their code quality yes but must be held accountable by an external force. I've been in too many organisations where empty, useless unit tests are generated to build coverage and say "look we are accountable" when they are not really. That might be an external QA / review team under a completely different management tree to the development team.
Realistically I've never seen an org where it's possible to scale it up and self-accountability being viable. There are too many defective personalities to manage this.
> Several strategies are getting deployed — shift left (e.g., more unit tests), shift right (e.g., test in production), transitioning from manual to automated testing and even removing distinct tester roles and trying to hold developers accountable for the quality of the code they are writing.
>
> These testing strategies are met with varying degrees of success and can even differ from team to team based on skill level, maturity, and culture.
AND (indeed) have QA done by someone with different skills. Development requires a focused mindset, in the zone for long stretches, int the box. Test requires distraction, exploring, out of the box — "hey, that's weird" or "I wonder what will happen if I do this".
Pretty sure that bug-free code is one of the prides of programmers worth their salt; but there's no helping, when you forget about something while coding, it is very likely that you also forget to test it. That's why we need a separate tester.
https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste...
In complex codebases it's easy to miss things when making changes and unit tests are great ways to catch issues in really simple bits of logic that have just later been overlooked.
Integration tests are typically so heavy and multivariable they can only be used to validate the most common scenarios. They're great sanity checks, but provide terrible coverage whenever there exists nuanced logic.
Indeed, that is exactly the point of unit test and code coverage: proving that your code is well organized and testable through its API.
> This is why getting artificial intelligence projects to create basic tests isn’t the impressive accomplishment people seem to think it is.
Also correct. Additionally, the point of unit tests is to document, to show the code's use case. It's generally not possible to device intent (why) from implementation (how), but if the AI were magically able to deduce that that's a sign that the code is either trivial, or plagiarized (or both).
> By embracing a more holistic testing approach, teams can deliver better software faster.
Sure, if velocity is your objective. Another approach is to stop and think about whether you're testing in internal (unit test) or external (integration, acceptance test) quality. As a developer, I use unit tests primarily to verify complicated business logic to prevent the embarrassment of obvious flaws in my code further downstream.
> Indeed, that is exactly the point of unit test and code coverage: proving that your code is well organized and testable through its API.
You can have 100% unit test code coverage and your code can still be absolute sh*t and do absolutely nothing useful. Also, your code can work, with 100% unit test coverage, and have the tests test nothing useful. So reasoning about the quality of your code through the quantity of your unit tests is highly suspicious.
And btw, TDD does not advocate code quality and well organized code - if anything, it lends itself to the opposite.
Totally true. But give me those unit tests, and I can tell that the code is sh*t.
If I have trouble understanding the source, it could be because it's doing complicated stuff. It could be highly optimized. It could I'm too dumb.
If I have trouble understanding the unit tests demonstrating what the code does, there is either something wrong with the tests or the implementation. Fix it, rinse repeat.
I wasn't saying that unit tests or code coverage are a panacea or that TDD guarantees great code. But it's a tool. It's better than writing documentation, but it serves more or less the same purpose. Sometimes it's useful, sometimes not. And its complete absence is usually a bad smell.
I think Michael Feather's book demonstrates how code improves by simply making it testable https://www.oreilly.com/library/view/working-effectively-wit...
For example, when writing a library that dealt with JWT and called into some other libraries, I opted for a lot of unit tests, I think the code coverage was around 90-95%. Writing tests for almost all of the functionality helped me actually discover one or two bugs which I could immediately fix without having them be released. Also, tests acted as documentation there, showing various use cases for the library, in addition to helping me actually make the API more user friendly.
Integration and UI tests are also useful to see how the solution fares in the real world, but once you have to automate a browser or deal with mocking the DB (if you can't spin up a container, migrate the schema and seed it with real data for your tests), the complexity skyrockets. You might still want to have them and plenty of them, but at the same time you'll deal with slow and flaky tests more often, especially if the tech you choose or are stuck with won't support your needs all that well (e.g. Selenium). It's even worse if the thing you're testing changes a lot.
As long as you have a staging environment, you should be even well off for anything from load testing to even the occasional bit of manual/acceptance testing, or trying to reproduce issues there. With the help of logs/metrics you can reason about changes/bugs/reproduction scenarios more safely, without impact on prod. Even better with other supporting things, like feature flags. Test whatever you can: sometimes the limiting factor will be time, other times the complexity of the system (or how testable it is in the first place, e.g. no fixed element IDs in a web browser), other times you won't get the infrastructure, but it's still a nice pursuit.
Types however, are not a substitute for unit tests. No type system is going to tell me if my income tax calculator covers all the edge cases.
All tests should be on business logic - code logic might not fit the business case at all and so that should be a fail.
That said, I don't follow what you mean by "to test is business logic, not code logic." To me, that is a distinction without difference.
An income tax calculator in a (sane) tax system is essentially a set of pure of functions or classes. This is a perfect place for a unit test. I might have additional unit tests for sub components of the hairy bits, but it's still unit tests all the way.
I fail to find a better example (complex functions of high arity with parameters from the same type ?).
If you use a language like Python, having tests that pass things that aren't the type you are expecting are useful. Likewise passing nulls to a nullable parameter, or different objects to an Object/Any type (or even a different implementation/type to a supertype parameter).
Those tests are part of the unit test repertoire, but unit tests should cover both expected cases/behaviour and unexpected cases.