Stop mocking your system

True "unit" tests are:

* faster to run

* give you less confidence in the correctness of the system (per time spent writing them)

* when they fail, give you more information about where the failure is

The more integration-y/e2e-y a test is, the more it strays from this: slower to run, more confidence that the system is correct, less info about where the failures are.

I think people have learned to undervalue the properties of integration tests and overvalue the properties of unit tests. Is it nice to know exactly what's broken based on the test failure? Sure. Is it _as_ nice as having confidence that the whole system is working? Probably not, in a lot of cases.

(I don't think there's general rules about which kinds of tests are easier to write. Sometimes setting up a real version of a component for test is harder, other times setting up a mock version for the test is harder.)

This blog post has tell us that we shouldn't use mocks but should instead "send data" to the code we are testing. I think this is supposed to mean using dependency injection instead. But the case isn't really made. Instead, with a waving of hands and wild assertions such as that I'm lying to myself, I'm left wondering what I just read.

The previous post was on cargo cult programming. It warns that CCG is bad but can't seem to define it. There is "how to spot" advice but no evidence to show that the advice works (except for a minor appeal to authority).

Yes, English is probably a second language, but the writing has more fundamental problems than that. The author needs to consider what a thesis is and how to support it. And after that, who the audience is. Etc.

Why stop there? Why not delete your test methods too and just test in production?

Mocks are just test code, same as your test functions. And they’re necessary fur unit tests. If the thing you’re testing talks to another component without a mock, it’s now an integration test instead of a unit test.

Unit tests test the API surface of a component. They’re useful for ensuring a component adhere to its documented API contract. It’s also useful for testing how it reacts to edge cases. Integration tests are useful for ensuring components work together, and testing that data flows through the system properly. But integration tests make it harder to test various edge cases.

For example, if my component talks to yours and issues several requests to your component with callbacks, it may be that 99% of the time your component calls the callbacks in the same order. So I can’t really validate in an integration test that my component works properly if the callbacks are involved in a different order. By mocking your component in a unit test, I can test any order I want.

Unit tests also tend to be more deterministic. By mocking, I can eliminate sources of non-determinism from outside my component. As a trivial example, my component might need random numbers, so I might mock the random number generator to return a specific chosen sequence that I know will exercise different code paths.

Really the lesson that should be taught here instead of “don’t use mocks” is “unit tests aren’t sufficient, write integration tests too”.

The author seems to believe people either mock everything or don't mock anything. Obviously using mocks for all your tests is a very bad idea, but that's not how things are done generally.

Unit tests allow you to validate a unit's behavior very quickly. If your unit test takes more than 1 second to run it is probably a bad unit test (some would argue 1/100 second max so your whole unit test suite can complete in a few seconds). In unit tests you use mocks not only to keep the test hermetic, but also to keep the execution time as low as possible.

Then you should have integration & e2e tests where you want to mock as little as possible, because you want a behavior as close as production as possible. For those you care less about how long they take. That's because you usually don't run those tests at the same stage as unit tests (development vs release qualification).

The author does not make the distinction between different types of testing, the resulting article is of pretty poor quality imho.

I agree with the premise of this post. Instead of mocking, design your components in such a way that they are easy to spin up in tests. Or use containerized versions of services.

I started disliking the idea of mocks a few years ago. I was writing a system based on the Play framework. Play framework used to (still does?) come with a dedicated integration testing environment. The problem with it was that the setup of the testing world slightly differ from the real world. I was bitten a few timer by the real world setup process while the tests were executing flawlessly. In essence, there was no simple way to test the real world construction before deployment to production.

Since then, the only integration tests I accept as real tests are the ones that test the production code path. Database? In containers. Kafka? In containers. etcd? In containers. There are exceptions, though. Proprietary APIs like SQS, or Kinesis, or Firestore are the difficult ones. I usually replace them with hand written memory bound implementations with an abstracted API adhering to some general top level constraints (put / get / consume / publish). This does not prevent errors rooted in the wrong understanding of the design principles of the dependencies, for example, consistency guarantees or delivery ordering, but those are usually possible to cover with additional tests further down the line.

> But dude, I don’t want to use a real database (or AWS endpoint or rocket launcher) in my tests. Debatable, but fair enough.

Is this debatable? Hermetic tests give you a lot of things for free and I don't really see a reason why you would default to making all of your tests hermetic.

The real thing that this article is touching on is that your tests should test all of the code you write but not code others have written. This sounds wild at first but: do you need to test that Postgres knows how to parse a query? Probably not. I think the postgres team knows how to test and release their database and I don't really need to spend time doing that. Then, the next layer of abstraction: if you use an ORM or some middle layer that abstracts the database do you need to test that the ORM knows how to talk to postgres? In a unit test, likely not. The ORM people have provided you an API and you should use their API to fake/stub/mock/whatever that system so you can focus on testing your business logic. After you have that system built you should then build integration/E2E tests that actually talk to hermetic copies of the real systems. An easy way to do this is to build a troubleshooting cli tool that you can run against your backend services/dbs/etc that can be used in CI against a copy of your backend or in prod to debug configuration issues.

The mocks are the symptom. The problem is that your code doesn't restrict side effects in any way. And so you end up with integration tests for everything and setting up a single test requires recreating the universe from scratch and slightly tweaking it on every run.

But that's what our program does, it talks to databases and file systems and HTTP servers!

Sure, and the effect of doing those things is moving data around. What does your program do with the results of these side-effects? Does it parse it? Transform it in any way? Decide whether to run effect A next with the result or effect B? This is the code that, if extracted, can be tested in isolation of databases and HTTP servers. You want as much of your code to be in this place as possible. It's a thousand times easier to test and the tests are thousands of times more reliable because they don't perform any effects.

Mocks have their places but if you can't test your code without mocking out the universe then the problem is that your code is interleaving too many effects with the core logic of the program. The cure is to refactor effects to the edges of your program and run effects in one place in your code. Make the rest just plain, pure code as much as possible.

This isn't a terribly practical article. I don't disagree with mocks being an "alternate reality". The author is entitled to their opinion on whether this is a good or bad thing. This said...what is the alternative? Integration testing all the way down?

The implication here is to work with stubs over mocks (i.e. I need to work with S3; I would then abstract that to provide a StubObjectStore to replace the S3ObjectStore used by other pieces of my code during tests). Great; I know they work now. But at some point, I need confidence my S3ObjectStore handles everything correctly. Do I give everyone on my team a bucket? Perhaps their own test AWS account? Test it, but only in the pipeline on its way to an intermediate stage? I can't control how AWS writes their SDKs (spoiler alert: they don't stub well), but I need some confidence that I can handle their well-understood behavior that scales. Likewise, I often can't control the libraries and integration points with other systems, and mocking offers a "cheap" (if imperfect) way to emulate behavior.

Mocks make me sad. They are so often misused by people with the best of intentions. I have seen so many tests that literally only assert the mock expectations and say nothing about the code under test. I have watched upgrades and refactors take 5x longer because someone steps on a landmine of overmocking. It is so common to see people mocking dumb data objects, containers, and pure functions - creating Frankenstein widgets that break all contracts the original had except for the 1 code path the original developer used 6 months ago. I've listened to DRY/SRP fanatics defend mocks until they realized that their tests were littered with unreusable, low fidelity copies of their production system in the form of mocks.

Mocks are a power tool and you can use them to do things that they are not the right tool for the job.

Another fanatic blog post about how something is always correct and another thing is always wrong.

Blog posts like this will lure you into thinking that there is a single right approach. Don't fall for it.

Do what makes sense. If it doesn't work try something else the next time. Becoming good is about growing your ability to make the right calls, not blindly following a methology.

One of the most obvious problems with mocking is that the team that develops some code usually also develops the mocks that are used for testing it. So precisely the same misunderstandings will be present in the code as in the mocks. In other words you are not really testing anything.

From my experience most errors are at boundaries between code from different teams. Mocking does not help here.

My favorite form of tests are what I usually call subsystem tests. Try to test as much code as is feasible with each single test.

Usually there are parts of your system that can be expensive to setup or use. For example, creating and filling a real database can be slow. In this case you could use an in memory database. Creating it and filling it with some representative data can be very fast. This database could be used by multiple teams, and is vastly superior to mocking.

A similar approach can be used with other expensive parts like remote procedure calls [0], or input from browsers.

This approach works when you design your system so that it can easily switch between using the real (expensive) resources and the ones that are only used for testing. But that is not very difficult.

[0] Both REST and SOAP are RPC

I've generally had a little bit more success with mocking when I'm hiding that dependency behind my own interface. So for example in Java, instead of trying to mock the AWS provided class, I write my own class (like a facade or repository pattern) which has a very simple interface of a success case and maybe a couple of relevant failure cases. It's calling the AWS library within it. But my mocks are at the level of my facade class which I find easier. The drawback is I'm not sure if there's a good general strategy to test the implementation of that facade. Most of the times the implementation is simple enough that I can do some simple integration tests for the most relevant cases, but there's always a risk that I am missing out some weird edge cases and I don't know how to properly deal with that.

> QA says: “it doesn’t work”. Dev says: “it must be working, all the tests pass”.

I've never experienced this, and I'm kind of doubtful this is a true reaction. I guess maybe it could happen in a team that is totally dysfunctional, where there's zero trust between QA and developers .. but in that case mocks are not the problem.

The realistic reaction is "huh, guess we are missing some test cases".

Ghost of Christmas Future:

I did some contracting work for a company with a 1.5 hour test suite that ran on every deploy, the section with mocked tests only took a couple of minutes — the rest were end-to-end (no mocks).

The worst parts of those non-mocked tests:

* They would interfere with each other, and could not be run in parallel.

* They were subject to real-world variability and were not entirely deterministic.

Management wouldn’t budget fixing or replacing the tests. Bugs still regularly found their way into production; arguments to pare down the tests were vigorously rejected.

My takeaways: If possible, use a language with a strong type system to avoid writing as many tests as possible. Move as much application logic as you can into pure code (so it can be tested in isolation). Observe the test pyramid.

> All of us have used mocks in our testing code, to substitute the real things. I know I’m definitely guilty of this. Why though? Why do we do it? Well, It’s usually because initialising those collaborators is not trivial (i.e. can’t be done with a one-liner).

I practically never mock something because I’m concerned that creating the real instance is more than a single line of code or non-trivial.

I don’t want my code making network requests if the actual instance is some kind of network/service client. As a matter of fact, I don’t want network requests in my unit tests. At that point, they’re not unit tests, and they could fail because the request failed or the system on the other end had issues - unrelated to what I’m actually trying to unit test.

I think the author fundamentally has a different idea of unit tests. I write unit tests and mock some dependency because I want to test some code in isolation where the “mock” is some interface with some possible inputs, outputs, or it fails perhaps by throwing an exception. And I can set up this behavior and validate my code gives me the expected outputs or makes some function call on a dependency.

The blog author seems to be conflating unit tests with integration tests, and I’m even wondering what’s the point - if I take them as objectively onto something, perhaps we should throwaway our tests and just test in production.

Sorry, this is bad advice. If you test against 3rd-party systems, eventually your system will grow to the point where you are spending a good deal of time dealing with your 3rd-party's random breakage (downtime or flakiness rather than API changes).

I learned this lesson twice the hard way: once with social media networks like Twitter and Facebook, and the other with Google Drive.

This again. It's about balance, people. It's not binary - complimentary unit and integration tests make for an excellent measure of confidence.

Docker has made most mocking nonsensical considering how easy it is now to use the real thing... But I would disagree with the premise of mocking as being non-starter. Often times you want to unit test and don't really care if you're using the real "thing" but want to hit code that's got zero to do with that dependency. Good example, we use Okta to authenticate... We want to run unit tests that test how a component in our UI works within our application, we mock Okta to get around our authentication for testing that very thing. When we want to test authentication with Okta, that's what we do.

I feel this can be summarised as "integration tests > unit tests". And yes.

However, for tests you want to run locally and quickly, the question should be, can you get the speed advantage of unit tests with the largest possible amount of integration? Can you only mock things that are "outside the system" (such as DB, networked services, and file system)?

IMO the biggest risk of integration testing is non-cleanup. I feel that is one of the major positive use cases for mocking. A mock DB will not retain garbage from previous tests, or previous runs of the same test.

Mocks and fakes are used to test your code under isolation - like an experiment, only one variable should be under test at one time.

In addition, most stateful code that can fail, does so intermittently, is susceptible to load delay at scale that can effect your customers and your wallet, may not be at the released version you expect, requires clean-up so that your storage isn't clogged with useless test data, which may or my not be mocked, effects downstream analytic services, etc.

Isolate your unit tests with fakes and in-memory test-local storage, run proprety-based tests with generated data for both inputs and returns that confor to the constraints of your domain, including possible expected error conditions, and unit test in parallel. If you do these things, you can and will have many of the benefits of integration tests, and won't waste time writing carefully crafted test data/scenarios.

Integration test when you get a bug in production, and run the integration tests in an isolated environment identical to the one you deploy to prior to deployment to a place where your users will be interacting with your code. Cleanup can then simply be destruction of the test environment. There was a talk on blue/green deployment the way AWS SRE does it a few years back that was great. While I have the notes, I can't find the talk, but it's a much more complex process than running tests against your up system.

I used to use mocks an awful lot more than I do nowadays. I learned to do that style of testing from the book Growing Object Oriented Software, Guided by Tests (the "GOOS" book, which is still well worth a read, even if you don't subscribe to that style of test driven development). I'm still of the view that mocks are extremely useful if a) you're working in a highly object oriented style (where systems are composed of objects that communicate with their collaborators by method call) and b) you're unit testing relatively small units of code at a time. Mocks are very useful if it's important to you that object X must call object Y, do a computation and then call object Z with the result.

There are two key insights that helped me to eliminate the need for so many mocks: a) modern languages that support functional programming allow you to separate the concerns of computation and interaction with collaborators, and it's an awful lot easier to test drive a pure function that it is to test drive a graph of collaborating objects; b) modern hardware is sufficiently fast that it's much more feasible to spin up a whole service (or many services) in order to run tests on them, and you don't need to write fine grained unit tests merely in order to make the tests run quickly enough.

> And guess what… you have to keep the mocks in sync with the real things

It is a valid problem but it's also possible to write mocks that are small and simple enough that this becomes trivial.

In my experience when testing a module necessitates a complex mesh of mocks and an intricate knowledge of inbound dependencies than this means that the problem are not the tests nor mocks but rather a problem tight coupling. The tests are simply telling us that in an unexpected way.

My rule of thumb is that if I can't mock it with a "every thing at default except what I want to test" than there's something wrong with the code, not the mocks.

And most integration tests can be eliminated by refactoring the code with proper contracts so that If A depends on B which depends on C then testing that A, B and C completes the chain of trust from A to C and doesn't necessitate an integration test from A to C.

> What you almost always want, when mocking, is really just different input data for your module

>What you almost always want, when mocking, is really just different input data for your module, data normally provided by a long stream of collaborators. Just build your program in such a way that you can send it this data, regardless of the runtime (unit test framework, test env or production env). Yes, it is possible and highly desirable and don’t be in denial right now.

Mocks are data in a nice, encapsulated and understandable form. I'd rather have a simple mock than "data provided by a long stream of collaborators". I can't recall how many hours I've lost by trying to reverse engineer huge pile of data and configuration files because the "tech lead" wanted a "life insurance" to make sure we broke nothing.

I recently wrote an eerily similar piece. My focus is more on unit test but the idea is the same: https://qvault.io/clean-code/writing-good-unit-tests-dont-mo...

Strongly agree, especially when it comes to things like AWS services. Their APIs and services evolve so quickly that things like local mocking (or emulation for local development) is an anti-pattern.

Where possible, I prefer to utilize short-term, pay-per-use infrastructure for development and testing.

If the system doesn't work but all the tests pass, then you're missing one of two things: code coverage, or articulated requirements. Either code that you're not testing is crucially responsible for the system outcome, in which case it should be tested; or the demands of a unit of code with full coverage and passing tests don't include something that a consuming unit needs it to do.

Neither of these conditions is an argument against unit testing or mocking. Both indicate a fairly basic gap in how you're testing that should be fixed directly. If integration tests would catch this gap, you're still missing the basic thing you need while incurring the cost and risks of integration tests.

I try to adopt the approach of using a functional core with an imperative shell (even if the latter is handled in a OO fashion).

This way, I can unit test the functional core without the need to use mocks or stubs, and rely on integration tests to test the imperative shell.

I agree with the author. Some devs take unit testing too literally. I have come across tests which mock every units that a component being test depends on. It's a terrible practice. First mocks are hard to maintain and second over time this friction generate disparity between real code and mocks. I have seen tests that pass even when the actual implementations have diverged.

The rule of thumb is, if you can test without mocks don't mock.

Most of the tests we write are integration tests not unit tests. Even at function level, a function being tested calls multiple other functions to get the things done.

My experience with mocking was sad.

I had unit tests; they all passed.

I had end-to-end tests; they all passed.

In production, I had about 1 in 1,000,000 failures. The most likely culprit was broken TCP connections, in turn most likely due to the enormous network load I was generating, which was only tolerable during rare windows. I tried mocking Linux TCP socket behavior, but failed to find a formal descriptions of the possible failure behaviors, and had to guess at a lot of things; I ultimately didn't finish before the project was abandoned.

I prefer to mock everything, with one exception: Refactoring old code that wasn't built with unit testing in mind.

For this, I'll go with the "social" unit tests up to the point where the code calls an external service or crosses a network boundary. I'll always mock these things because they slow down your tests massively. A 500ms external service or db call might not sound like much but multiply that over 5000 tests.... I'll still try to mock whatever I can that can be mocked with minimal effort.

what the author is talking about in a short rant is classical vs mockist TDD.

for better content on the subject:

https://martinfowler.com/articles/mocksArentStubs.html

https://martinfowler.com/articles/mocksArentStubs.html#Class...

https://www.thoughtworks.com/insights/blog/mockists-are-dead...

"The classical TDD style is to use real objects if possible and a double if it's awkward to use the real thing."

I've been working in a classical TDD style for the past 8 years, after at least that many years of Mockist TDD. A code base built in classical TDD style is much easier to maintain and change, but it does require more test setup which can easily be pulled into re-usable test data scenarios etc. We'll use fakes for services that are external(S3, DynamoDB, third party APIs, etc), we'll use real DB code pointing to h2 instead of say postgres other than that there's no test doubles. I would not go back to using mocks by choice.

I have seen some very very bad tests written with mocks. The tests were written in such a way that you couldn't change the underlying implementation without completely rewriting them because they were testing underlying implementation behavior down to ensuring that certain methods were called on certain libraries used by the underlying implementation code. Please don't do this. It is a nightmare to improve the implementation.

Recently, I thought about a process I call "Test Coverage Driven Testing". It is similar to TDD, but more adapted to when we write tests after the code (you know you do too, at least occasionally, don't lie).

It goes roughly like this:

- write one integration test for the "happy path".

- using some test coverage report, identify untested cases.

- write unit test for those.

I find I helps me find a good balance between time invested writing tests and benefits reaped from those tests.

Do you have a similar process?

Developers end up hearing my admonition that their unit tests should not be uptime indicators for some third party API. Those external things should definitely be mocked.

Whenever/wherever there is piece of code that can tested as part of a unit test, write a simple test or two for it.

However, each time I run into 200 lines of setup code that will make the test pass trivially, I think an opportunity to simplify something was missed.

This resonates with me.

I'm not at all an expert in testing; but I instinctively take the attitude that every line of test code is code that itself has to be tested, even though it's never going into production. I don't entirely trust that instinct, but I'm suspicious of test rigs and mocks. But I don't know how to test a "module" without them. Que faire.

As the blog post says, sometimes people mock because they're incapable of running the real system. That's just another reason why it's important to be able to run your system... http://catern.com/run.html

I would make an exception for SQLite. Here's how you do it:

- Write your schema so it's compatible with SQLite and your production DB, such as PostgreSQL.

- When you're running tests, use SQLite instead of Postgres to run those tests.

- Use SQLite in production. Go ahead, do it. You don't need Postgres, SQLite is fine. I promise. Trust me

Regardless of what this article says, mocks are really useful when your project is partially written and you want to test the pieces you have in hand. Mocking then is absolutely the right approach and might entail mocking many things (which will eventually be replaced by working code.)

I oppose to that. We have unit tests calling API endpoints of third party sandbox. Once a week all of our CI fails cause third party sandbox is down

If your application has well defined boundaries with clear contracts for communication - mocking is honestly a great strategy.

Shameless plug. Speedscale.com (S20) let's you test with real world data, without the mocks and fakes.

Stop telling me what to do.

(Using imperative tense when presenting your ideas to strangers is a really douchey thing to do)

Absolutely terrible advice. No, I won't have my CI rely on every integration point in my application.

Serious question. Do most test suites not allow you to run a subset of tests?

Write tests. Not too many. Mostly integration.

(ps. Mock only when necessary)

This has served me quite well.

Two words:

False dichotomy.

Nothing should ever be mocked. Period. If you don't agree you likely don't understand many things. Think about why mocks exist in the first place. Mocks exist for things that can't be tested. What you are doing is creating something completely new in place of that thing that can't be tested and testing that new thing instead. Utterly pointless. If it can't be tested, you can't test it period. It's like saying drugs can't be tested on humans, so you create a plastic dummy in the shape of a human to test the drug on instead. Come on man.

There are books, there are experts, there are people with years of experience who think they know what they're doing but the minute I see a mock in a code base which is probably 99% of what's out there I already know that the people who designed the system don't know what they're doing.

Nothing. Ever. Needs. To. Be. Mocked. Many of you are thinking you know better. You don't. Mocking is bad. Allow me to explain.

There are two parts of your code. Code that can be unit tested and code that can't be unit tested.

Code that can't be unit tested is simple. Any code that has to touch IO can't be unit tested. Period. Any code that doesn't touch IO can be unit tested. It's that simple.

Why do people mock? Because people write systems that are too tightly integrated with IO.

Imagine this function:

    function addTwo(x: socket) -> number{
         return socket.get_one_numer() + 2;
    }

Now you have a function that adds 2 to a number. But in order for it to be unit tested you have to Mock the socket. The socket is a parameter polluted with IO. If you have that parameter touch any part of your code then all of that code cannot be unit tested anymore and you have to mock that socket if you want to regain unit testing functionality.

What's the simple way to fix this? Easy Keep ALL IO segregated from the rest of your code. Keep IO functions and methods super small. Do not inject IO polluted objects into other parts of your code. It's trivial:

    function addTwo(x: number) -> number {
          return x + 2;
    } 

    function getNumber(void) -> number {
           return socket.get_one_number();
    }

There. No mocks. addTwo is a function that can be unit tested and getNumber is an IO function that can NEVER be unit tested. That's it. No need to mock it.

There are two types of IO functions. Input and Output. Inputs have void parameters. Outputs have void return values. These are the functions that can't be unit tested. If you keep these functions super small and tiny, guess what? Most of your code can be tested with unit tests and you're golden.

Instead what you'll see throughout your career is typically this garbage:

    class RandomObject(int param1, Socket paramSocket, IOService paramIOService, LogService logService){};

or some other overly complicated, over engineered structure that necessitates Dependency injection or some other garbage pattern that forces people to mock things to test.

Think about it. Every single method you put in that class cannot be unit tested. By using this stupid pattern you pollute that entire class file with IO and nothing can be unit tested unless you mock the socket and/or the IO service.

The problem is, this pattern even though it's so obviously detrimental is used practically everywhere because the complexity of the pattern makes it seem modular and "advanced" when really it's just bad.

Additionally I neglected to mention that it's not only IO. But overly complicated logic sometimes is mocked as well. To that I say it's the same problem as IO. If you find yourself mocking overly complicated logic to test some portion of your code it means that portion of your code is too tightly integrated with the rest of the universe. You need to loosen the coupling.

Mocking has a place, which is not unit testing. If you find yourself mocking a dependency in a unit test you are not unit testing any more.

Those points of contact with 3rd parties should be clearly defined and encapsulated at the perimeter of your system. Mock at that level. Not when testing business logic.