Warming Up to Unit Testing
junglecoder.com
junglecoder.com
Without a suite of unit tests, there's no way to know that your refactor hasn't introduced bugs of some kind. A suite of tests won't necessarily catch every bug you'll introduce, but it will catch many of them. I've seen this repeatedly.
I've had many conversations with developers who claim they don't need unit tests. When I ask "How do you refactor?" the answers is usually some variant of "I don't."
That said, as someone who has spent the majority of his career writing dynamically typed code I can definitely recall far more instances where statically typed code has run and worked first time vs times when I've written some code with unit tests and it worked first time.
Use the best tool (type of test) to document & test the business or technical rules being requested.
Tests help teams (and future-self of the individual who wrote the code) understand why certain code was implemented & what purpose it is meant to solve.
As the business changes the rules they want implemented, having well written out tests seem to help significantly with handling those rules. It also helps when you want to refactor for the sake of code understanding & readability.
It is often the case where both developers & those requesting the changes forget why some of the rules were put in place to begin with.
When the layer is essentially pro forma and has no logic of its own, this is excruciating, but the line coverage number rules all.
Let's say you have controller -> Business logic layer -> Data access layer.
The controller takes in some input, validates it, passes it to the business logic layer. The business logic layer retrieves some data from the DAL, does some logic, maybe writes data back to the DAL, and passes the data back up to the controller. The problem is that the business logic layer is usually doing two things at the same time, in the same function: the 'plumbing' (calling other things and passing data round), and the actual business logic.
When you try and define a boundary around the code, it gets messy, because what you want to test, the business logic, is mixed in with the plumbing so you have to use either dependency injection or mocking to force a boundary. This makes your tests more brittle as changes in things you don't want to test (your dependencies) affect your tests.
If you pull out the business logic into its own modules, and instead have a function that orchestrates the business logic (i.e. tell the database what data you want, tell the business logic functions to perform an operation on the data), you end up with a nice clean boundary around your business logic, which can be unit tested easily without mocks.
You then test the 'plumbing' part by using a component test that checks that the inputs and outputs of the component as a whole (business logic + plumbing + data access) are correct.
But, I think this distinction gets blurrier the larger your organization. While you may have intended some class in your web app as an implementation detail, the larger your organization is, the more likely it is that somebody will start using that class for some other purpose. And thus and implementation detail becomes an interface that other people depend on.
Agreed. I think that in many cases it is a signal that your code is too complex, and may be violating the Single-Responsiblity principle.
Start off with things clean, consistently try to keep it that way, and it's actually pretty easy and not too much effort.
Start off with a mess, or start off clean but then neglect things for a while, and you will wind up at the bottom of a hole that it is going to be a long, painful slog to dig yourself out of.
This applies to tons of things: coding, keeping your bathroom clean, car and home maintenance, personal relationships, financial management and record-keeping, etc.
Seems to me writing like this is best served with a few book recommendation tweets and doesn't really make sense in this format.
Why:
Mostly, I think it's down to having a better understanding of where to try to break dependencies, specifically trying to abstract away from the outside world (akin to how they did it in the Moonpig billing system).
The how is still in progress. Perhaps the most notable instance I can think of it is that recently, due to work on breaking dependencies, I was able to stub out a RabbitMQ connection for an in-memory class, so that I was able to easily test how two components interacted without having to stand up an external service.
For me, looking back, I think it was the Moonpig article that's had the most influence on how I think about things.
Do you have any other questions?
This was a good way to get our coverage up. Something about writing the ‘test’ in the context of the function makes it seem easier. Also it helps that you get tests + docs + intellisense all in one fell swoop.
Maybe I'm doing it wrong, who knows...
(1) You're working on a system that is poorly structured for testing.
(2) You're trying to make a unit test do an integration test's job.
Both are incredibly common. In particular, if you find that you're spending a lot of time creating mocks/stubs to support unit tests you'd probably be better off finding a way to inject errors in an integration test. Error injection can be done at any level, works even when the code structure makes mocking difficult, and exercises code paths that cross multiple components in their "final" close-to-production form. IMX most of the people who go crazy about unit tests just haven't thought deeply enough about the problem to realize that error injection is strictly better (and a lot easier) most of the time.
Mocking is for exercising all inputs to a method, not just error injection. In fact that's fundamental to unit testing, and a necessary part. And it can grow to become a nightmare.
Only in your head. I'm not trying to claim unit tests aren't useful, or that any particular unit tests aren't good. The only fallacy in sight here is strawman.
> Mocking is for exercising all inputs to a method
How useful is that if they're values that the real version of the mocked component can never return? As I said right at the start, this kind of validation has value when changing code later, but that's a lower priority than validating that the whole system as it is today does what it's supposed to.
Doing unit tests right is valuable, but is much more of an art than most people realize. A good unit test should cover inputs and paths well, but not constrain implementations. Most unit tests I've actually seen - quite a lot BTW - are the exact opposite. People need to stop thinking in terms of depth (adding mock after mock and implementation-specific check after check for a few obvious scenarios) and think in terms of breadth (lighter weight validation of more scenarios) instead.
P.S. If your "inputs" are results from functions that have to be mocked for testing, it's often a sign that the code has a backwards control flow. When that happens, it's better to fix the control flow than to add more mocks.
Does this not give credence to the notion that there may in fact be something wrong with unit testing in general? Is it not just compounding the problem?
I'd rather spend my limited cognitive currency and time in the system itself, not in mastering the art of unit tests... Unit testing should be a tool not an art. If a tool doesn't simplify my life, it has limited utility.
> I'd rather spend my limited cognitive currency and time in the system itself, not in mastering the art of unit tests
That's not quite an either/or. Over an entire career you'll probably work on many systems each requiring their own specific knowledge. Knowing one system is great as long as you're working on that system, and mostly useless thereafter. Knowledge of how to write good tests (unit or otherwise) is much more transferable. Every team can use somebody with those skills.
Over the years I have come to reject some things that are seemingly fundamental to software engineering in the industry. One of them is unit testing. I don't believe that unit testing is harmful in itself, but excessive use of it is a sign of weakness in another area.
- if your code relies on unit testing for correctness than your coding methodology (or language) is error prone. A good programming methodology where the domain is well understood usually does not need a single unit test.
- If your code base utilizes dependency injection to make the code more "testable" with mocks, you are making the code 10x worse. There are other ways to make your code more modular without injecting new scope, logic and apis into an object.
This logic sort of applies integration tests which is basically testing the same thing as unit tests but touching functions that have IO. However it is a little different. If your program does Not need unit tests but needs integration tests it means things like:
- your type checker does not extend across systems
- the programming methodologies become more error prone as you move to other systems and you have no control over it.
- you lack understanding of the foreign system.
The thing with integration tests is that all of the above is largely inevitable and integration testing really becomes the only way to correlate (not verify) the entire system with correctness when you lack the integrated approach of a single programming language. Therefore because of this I am not against integration testing.
I recently green fielded a small micro-service at my company and I did not write a single unit test. It had one logic error and that was largely due to a flaw in the type checker. I can imagine many engineers being aghast at the whole concept.
So essentially I've had the opposite conclusion. Unit testing to me is a symptom of bad engineering practices. If you rely on it, something is wrong.
Tests add real value for me. They are now usually the first "user" of my code. I often find design issues when writing tests. They also catch bugs (rarely but often enough to be useful) and make it easier to refactor with confidence. YMMV!
Most of the code I write doesn't need unit tests (automated or manual) because the way I programmed it reduces the complexity of the program to the point where my intuition and the type checker alone will cover correctness.
Basically another way to put is... the code doesn't have any errors in the first place so any form of unit testing isn't really needed.
Note that I am talking only about unit tests or tests that cover pure internal logic. Anything that touches IO or anything external becomes an integration test and That is not what I'm talking about.
I would try coding some solution in haskell with minimal IO. You should only need the type checker to verify correctness.
I don't know that I'd word it exactly like that, but one thing I've found over the years is any tests we have that rely on mocks basically just don't test anything at all.
If your whole test is "This gets called, then that, then that", you aren't actually writing a test. You're repeating what the code already says.
But your philosophy is correct and we are in agreement on that.
The thing I was referring to is the abstraction of dependency injection itself. It leads to code that is over complicated as the abstraction itself is not only pointless but adds unnecessary layers of complexity.
There are ways to target the philosophy of what dependency injection for unit testing is trying to accomplish without introducing the side effect of unnecessary complexity that comes with dependency injection. It's as simple as just separating IO functions from core logical functions. Don't bundle these two things together into an object or a single function, that's it. (Also don't use objects period, but that's another topic.)
I find it natural and convenient to think about "What dependencies on other parts of the system does this code have?" Expressing those dependencies explicitly feels like it reduces complexity, not adds it.
But, I'm not talking about pulling in a big fancy DI framework, just making dependencies explicit in your function/class parameters.
I will say that DI is sometimes used as a tool in overly-abstracted systems. An example that comes to mind is the ASP.NET MVC framework -- with DI and inheritance, literally any behavior can be overridden in fairly opaque ways. Trying to suss out the concrete behavior details is like swimming in quicksand. (Or it used to be that way, haven't touched ASP.NET in a while.)
As an aside -- I'm curious about your programming language of choice. I think DI is a lot more useful in some PLs than others. For example, I find JS code often uses imports to create complex graphs of implicit dependencies, and DI can help tame that complexity. But for other PLs like Python or Clojure, I basically don't use DI at all.
Now Java and C# are different because they are compiled. You need a DI tool to do dynamic dispatch if you want to mock.
You can't inject a nuclear reactor into a burrito in any language untyped or not because it will lead to an error.
The only difference is python/js the error will happen at runtime while in C$/java the error happens at compile time.
The main difference is the type. In typed languages you need describe am interface or a class of types if you want to do mocking or dynamic dispatch while in dynamic languages you don't need to explicitly define this as a type, the definition exists in how you use the object that is passed in.
Either way DI, however universal, in any language is bad practice.
I find it valuable at times, depending on the context, but I agree that extensive mock testing like this can get quite useless real quick.
Yeah, mocks are just a way to zero out the dependencies so you can focus on the unit you're actually trying to unit test. If all you're doing is zeroing out dependencies, you're not testing much. Every mock you bring in carries and assumption, and those assumptions need to be tested, too.
Cases like that really call for some small scale integration unit tests, where you actually bring in your dependencies and test them all working together.
EDIT: What I'm working on currently doesn't have a hardware component to it, so this might not be practical in the future
It's easy to write something small and correct from scratch (like a greenfield microservice). It's far harder to maintain that over several years and several changes of hands.
If your coding methodology was good, the type checker is enough to verify these contracts.
A language without a type checker will have more need of unit tests.
Type checkers do not verify behavior. Unit tests verify behavior
Types do verify behavior - e.g. this function always returns an Int, that function only returns a UserId, this other function doesn't do any I/O.
What you really want is to be able to state properties, e.g. "for all x/y, mul(x, 0) = 0, mul(x, 1) = x, mul(x, 2) = x + x, mul(x, y) = mul(y, x)", and so on. And working with these kinds of statements (via e.g. quickcheck or whatever) might generate tests, but feels a lot more like working with types. You're making proofs, not examples.
For me this is the best side of unittests: their validation of “correctness” is just one aspect, but their job as a guarantee of decoupling and a living documentation is at least as important. Even a test that is only written and compiled but never run provides great value for architecture and documentation.
If you want to write unit tests for documentation that's a different argument. I'm not against that.
Unit tests famously don’t prove correctness (the absence of bugs) only the presence of bugs, so I guess it’s wrong to compare them to actual verifying actual correctness.
Unit tests are like statistical samples of all possible test cases. So unit tests while they don't verify correctness overall they can correlate with correctness. That is why you only need a couple unit tests to correlate with. a degree of correctness with a domain that's almost infinite. The phenomenon that enables science also enables unit tests to work.
That being said, there are programming methodologies of managing complexity that negate the need for unit tests. This is my main point.
I presume you're talking about proofs, or maybe something like Idris?
It's really too long to get into but the programming methodology I'm talking about has to do with reducing complexity of your program to the point where your intuition along with the type checker should render unit tests mostly redundant.
For example a language with no nulls renders all unit tests that test for null runtime errors unnecessary. Add exhaustive pattern matching into the mix and you'll render All tests that cover runtime errors unnecessary because your program simply can't have a single runtime error (see elm.)
Both of the examples above involve either the language itself or a programming methodology or both. Rust and Haskell are two languages that force a lot of "methodology" onto your programming where a significant amount of unit tests that you would otherwise need in say python or C++ are rendered unnecessary.
Another way to look at it is using a highly restrictive form of programming. The more restrictive the less opportunity for errors to the point where your intuition and type checker alone should be able to allow you to forego unit tests.
There's more too it like segregating all IO from core logic but I've already written too much. Suffice to say what I'm talking about falls within a subset of the domain of a semi-popular style of programming. I don't bring up the name because that will only detract me from the main point.
Everything including nulls, index-out-of-bounds is unnecessary and can be elegantly handled by the type system.
But complex business logic like building fire codes, tax law, network standards, traffic rules etc aren't that simple. You are quickly into Agda/Idris non-mainstream territory if you want to represent some tax law in types. It's basically "if you have two or more dependents and make under 10k then the tax is 4.5% of the income over 5k minus...". It's of course possible to represent anything using a complex enough set of types - but I haven't seen a good example of a complex piece of domain logic that didn't need tests. I'm a big fan of e.g. "Domain modeling made functional" and similar, but usually the most complex calcultions are just that: calculations. You can encode things like "the input must be positive" or "the output is guaranteed to be positive" with types easily, but when you want to encode that "given the input 3 the output must be 16 on a thursday" your are beginning to probably see diminishing returns if doing it with types instead of e.g. property based testing.
Let me put it this way:
Given an exact set of tax laws, network standards or traffic rules I can implement this perfectly and easily with types, intuition and programming methodology alone. (Another hint: don't use mutable variables or even better forego variables and go point free; it eliminates a whole class of bugs that has to do with state.)
Implementing a specification is trivial even when the specification is complex. It's simply a mapping from english to programming logic. There is little complexity here.
I think what you're getting at is a sort of derivation of the specification from a macro goal.
For example, what is the best way to tax society so that society has maximal benefit and zero loop holes for bad actors? What is the best way to create a set of traffic rules and network standards so that throughput is maximized, fair among everyone, and (most importantly) 100% bug free?
This is hard and specs derived from a macro goal can be bad or wrong. This also usually happens before programming begins.
Bad specs are not something that unit tests are really used for. Unit tests are used for logic errors. To catch where a programmer implemented a spec incorrectly.
While I can see some logical value for using unit tests to test for macro level goals this stuff in practice happens at the IO level and is the domain of integration tests. You are not testing a "unit" when measuring a macro level goal, the philosophy of an integration test applies in this case because a macro level goal is basically operating at the integration level rather than the unit level.
Unit tests ensure that everything you thought about still works. Types ensure that ALL types are compatible.
What types do well is the everything case, you cannot forget something with types. However they only cover compatibility not correctness. You can have a type correct program that is wrong.
There are formal methods can can get to correctness. If you have the right constraints. The only problem is you can have the wrong constraints. Within that limit though it shows the program works correctly in all cases.
Unit tests prove only the cases you thought about work. While this is far below the infinite number of cases the above prove, they are still very useful because it is generally very easy to reason about the one case you do choose. If you choose the cases well, the majority of possible cases are "close enough". Thus a few well chosen unit tests can give you assurance that your code works correctly in the important cases.
For things that live outside of this domain the term integration test becomes relevant. If you are testing for unknown side effects it is no longer a unit test but an integration test.
This is not my point.
Unfortunately I can't go into detail about my point because it's just too long to talk about. I glossed over it and called it "Programming methodology."
In the context of correctness, with the right programming methodology, programming language and type checker you can use this and your intuition alone to forego the need for unit tests. This does not apply to integration tests.
Anyway without going in to deep the right programming methodology involves reducing complexity. It's similar to how if a programming doesn't have nulls you can't have runtime errors caused my nulls. If you use the right programming language or right methodology you can reduce much of the complexity of the program to the point where unit tests are redundant to your intuition.
Another good example of a language that pulls this off is Rust or Haskell. You don't have to go as far as COQ but the previous two languages force you into a methodology that ensures correctness to a degree that many unit tests become redundant.
Or to put it a different way, unit tests guard against an incorrect specification. Code can be proven correct and wrong at the same time. Unit tests make it much easier to reason about if what is specified to happen is what should happen.
As an aside, I'm not against blurring the line between unit tests and integration tests.
Dependent types and/or automated proof checkers should bridge the gap of intuitive reasoning. Basically the dual relationship where code verifies the type and dependent types verify the code should be strong enough to forego unit tests and guard against "incorrect specification."
>As an aside, I'm not against blurring the line between unit tests and integration tests.
Sure that's fine. But then the type of test and the context of what I am referring to is strictly tests that only cover pure mathematical logic.
Things outside of that bound cannot be covered by proof nor by programming methodology and are also not the type of test I am referring to.
- It's a must-have for business logic, because errors can lead to actual dollar costs (and looked at by the SVPs). Any maintenance change should require a unit test for this type of logic. This way our code confidence in deployment of a new feature or change goes up.
- Catching regressions. We've had errors we've missed in the past, and as our team changes and team members come and go, the want to make sure we are not making the same mistakes. Thus, automated testing on regression tests helps our confidence in code stability goes up.
- System failures. This is more obvious as an integration, but it's good to have these tests in place because once again they can be traced to actual dollar amounts. ALso helps confidence level of deployments by junior/new employees.
- Maintenance of legacy code over YEARS. I think most people skip unit tests because they are writing new code, which gets tested over and over during development, but adding a new feature to massive legacy code is still an aspect of our jobs. I don't remember how my code worked last month let alone 3 years ago, so unit tests help not only in confidence levels, but in speed of development because I wrap mocks around any services that I don't want to wait for or should not be triggering for this rapid-iteration development.
Each circumstance is different, but I think unit testing goes beyond just knowing the domain well.
Unit testing is by far the most valuable form of testing during development, in that it catches 90% of bugs before the code touches hardware. Also, after development, unit tests serve as regression tests that catch unintended changes made by others.
Mocking, whether or not through dependency injection, is absolutely necessary to enable testing when targeting bare-metal platforms, and very helpful to speed up testing when targeting an operating system. Dependency injection in particular usually constrains code to be written in a way that, although perhaps seemingly more complex at the surface, is a lot more maintainable long term.
Integration testing is also a very good idea. The thing with integration tests, though, is that they get bigger and more complicated and slower over time. Relying solely on integration testing will likely become a development bottleneck if your system grows in complexity. If you can run an integration test suite deterministically and efficiently on a development system, then the software under test probably isn't that complicated, relatively speaking. Also of note is the tendency for mocks and dependency injection to help out when integration testing as well; being able to completely abstract away hardware allows for faster than real-time deterministic simulation at a system level.
If you're doing well without unit tests and folk generally seem to be happy, then good on you. Unit tests are a pain to write. I'd suggest that you might be severely limiting your full potential as a software developer, though.
Maybe it's you who hasn't taken the red pill.
Safety critical software would be even more safe if you used formal methods. Formal methods verify logic to a degree of 100% while integration tests dig out unintended side effects or leakage.
Let me put it another way. In Computer science there's a field called formal methods. With formal methods it's possible to do things like dependent typing. These techniques and technologies can verify your program to 100% correctness. This is not unit testing to correlate correctness to a certain degree but actual 100% verification.
Such a technique has pitfalls in the sense that it is brutally hard. I am saying that there are programming methodologies that can be used to go above standard typing and below dependent typing where the program is not verified to 100% correctness but the correlation of correctness is so high that you can forego unit tests.
Some languages like Haskell and Rust already sort of enforce these programming methodologies / technologies on you such that most unit tests become redundant.
You need a lot less unit tests for a haskell program than you do for javascript.
Therefore a program that relies to heavily on unit tests is a sign that the programming methodology is flawed. Something is wrong.
I think that's being too extreme and dogmatic. Unit testing and small-scale integration testing (implemented as unit tests) are a way of testing your assumptions and documenting and communicating your assumptions to others. Both are good engineering practices.
You just have to be pragmatic about how you use these practices. Teams definitely get too dogmatic and narrow minded about mock-heavy unit testing, and some people do manage to write code without writing any unit tests, however those are both extremes that are best avoided or of only limited applicability.
The benefit of unit tests is that certain things are more economical to test that way. You literally can test everything though integration tests of varying levels (or manual testing!), but it'll take more effort and run more slowly. If you want to thoroughly test a regex, just test the regex by throwing a dozen strings at it. Don't bootstrap the program and construct valid input requests to test each of those strings (but do do that to test that everything can work together, which includes a subset of those strings).
Or even write the test cases separately, then import them into a unit test runner that can check them all very fast as well as an integration test runner that can check them all very slowly.
I use documentation for communication and documentation. Unit tests may do this but this is a side effect. The purpose of unit tests is to test your own logic and find mistakes you've made.
I'm saying there are ways to code where your intuition and type checker alone are enough to verify correctness without the need of a single unit test but with the same correctness as if you did verify your program with unit tests. You don't even know what this "way" is as I haven't even described it, so you can't even argue for or against it.
Dogmatic is a word for those who spout their point without understanding the reasoning behind the other point.
- If your code base utilizes dependency injection to make the code more "testable" with mocks, you are making the code 10x worse. There are other ways to make your code more modular without injecting new scope, logic and apis into an object.
Dependency Injection is one form of composition. One perspective to look at it, is that objects are like tiny scoped programs with variables and methods. To do a dependency injection is to put a program within a program.
Is this a good way to compose primitives? It's like parsing a python program inside a java program inside javascript. What are other ways to manage complexity and compose primitives without dependencies everywhere? What is a programming primitive? Is the most fundamental primitive of computing a function? Should we be composing bricks to form buildings or compose buildings to form bigger buildings?