Don't Use Mocks | Hacker News Reader

Don't Use Mocks

joeblu.com

72 pointsjoeblubaugh84 comments

Can we not have articles like this with blanket statements like this please? As with everything in software, no one size fits all. Mocks have their place. There are situations when the only way we can test something out is to mock out a certain function call. Sometimes our dependencies are so complex and deep that we cannot just replace it with a fake. But with a little mock we can replace a function or a class within that dependency to decouple our tests from it.

Having a title with a blanket statement like "Don't Use Mocks" is either plain wrong or clickbait. In this case, it seems it is both wrong and clickbait. Such blanket statements and clickbaits specifically trigger people to get into a long-winded debate (ironically, my own comment here is a case in point) about something that is obvious and otherwise uncontroversial.

I don't grasp the difference between a mock and a fake.

Both are substitutes for an expensive or unavailable component.

Maybe the fake is more dynamic than the mock?

The point seems moot.

Maybe more eloquently put:

https://martinfowler.com/articles/mocksArentStubs.html

Mocks, fakes, stubs. Every time I read this article I end up more confused than I was before.

I like Uncle Bob’s explanation:

https://blog.cleancoder.com/uncle-bob/2014/05/14/TheLittleMo...

That is a solid way to introduce the different test doubles. Thanks for the link. I agree, stubs and spies are my daily bread. I also use fakes very regularly. I hardly ever use mocks, for the same reasons in the post.

As a Python main I'm wondering if the difference has to do with a strictly typed language like the author's two examples, Go and Java, requiring more boilerplate and interface classes and so on, that in turn require more customization of the mock (since he writes that it "implements the API of what it replaces" which seems like it would be a strange thing to say about a python mock).

A fake is a 'local' or lower fidelity implementation of the real thing. A array/memory backed DB instead of postgres, etc.

A mock is a cache for a desired return value (which is hopefully what the real interface would correctly return).

Edit: actually sethammons link is great

Mock has expectations about how the function is called. If you read a file from a disk, and you expect it only to be done once, a mock is "usable" in this scenario to count the number of invocations. Note that there aren't outside, observable, state changes or behavior involved in this. It is about the non-functional introspection.

Fake is just a simple implementation, like in-memory db to stand-in for a real one. In the optimal case, provided by library authors.

I actually agree. I think the point the author is trying to make it that fake can be reused in more places and stay relatively same as the code updates.

But I personally think that instead of spending the time to write a fake, its better to just spend the time writing actual integration test with the real dependency (eg: just run the db in docker or something)

or if you don't want to spend that time, then just record the interaction (eg: db calls) and create "throwaway stubs". Use this stubs as long as they are relevant, and then generate new ones as your code grows. Save time "writing" any mocks, and you don't tend to couple too much with your mocks :P

> just run the db in docker

That creates unit tests that take ages to run - and fail surprisingly in hard to debug ways.

If running a micro service use sqlalchemy and switch your db to sqlite for testing purposes.

There are scalability limits to this but for most cases it works extremely well and is fast.

What he calls a fake is what I call a mock - although I don't think he's necessarily building a strawman here, there are developers who use mocks the way he's telling you not to, and they are creating the problems that he outlines. I'd change it from "don't use mocks" to "don't be stupid with mocks".

How I understand it:

A mock is a set of fixed data; sometimes people even load these from YAML documents and the like. They're often re-used for different tests.

A fake is a "fake" object created when needed, with the parameters you need.

Mock:

  user = load_user_mock()
  article = load_article_mock()
  run_test(user, article)

Fake:

  user = newUser(Name: "foo", Email: "foo@example.com")  # Or generate random data
  article = newArticle(User: user, Title: "bar")
  run_test(user, article)

The difference is somewhat subtle, but I often find mocks very inconvenient because you can't "just" change one without lots of stuff falling over. It's also much harder to do something "special" with them for that one test.

Thank you. Seeing this on page 1 is just sad.

Sure, the title is clickbait. But the message seemed useful: if your test rigidly requires every call be made in exactly the way it is made now, then your test will be brittle. For a less brittle test, re-implement the functionality in a simpler form.

What this article did not do was to describe the trade-off: what you give up by creating a more complicated "fake" (to use the author's term).

A naive question here, should I not then in a unit test, test that an API library is called with specific arguments?

I am confused, wouldn't using simpler and "broader" tests miss test coverage on e.g. specific error handlers?

Actually, I feel that making strong statements of opinion regarding code style is probably the best way to go about doing things.

The point is: if your tests just assert which methods are called on dependencies with what arguments (or something close to that), they are extremely coupled and brittle. Almost any change in the implementation will require changing such tests. And by their nature, they probably test some minor low-level things, that are not all that valuable to assert anyway. Mocks enable and encourage this kind of testing.

Better tests would assert some kind of higher level properties that are much less likely to change. This often involves making the dependencies you inject during testing some more complete simulations of real implementations. Takes more up front effort to implement them, but can often be re-used between many tests.

Tests on one hand help you modify software over time, but on the other hand increase maintenance burden. Being good at testing doesn't mean just writing lots of tests, but being good at judging the value of a test vs the cost of having it, which is very context dependent in itself.

This was an insightful comment. I shouldn't write tests that assert a method on my mock has been called, I should design a mock whose behaviour changes in response to interaction, and then assert that the relevant behaviour has indeed changed!

That change would still be the same amount of maintenance (if the underlying implementation changes, the mock will have to be updated to reflect that as well) but the test will communicate more clearly what the intention of the interaction is.

>This often involves making the dependencies you inject during testing some more complete simulations of real implementations. Takes more up front effort to implement them, but can often be re-used between many tests.

At higher levels, the up front effort is higher but most of that work that is much more reusable - reusable across implementations and different languages, even.

With tools like playwright and MITM proxy, the state of the art for these highly reusable tools has moved forward considerably in the last 5 years. I can now "TDD" everything in a way that actually makes sense on some projects - even a tweak in CSS (via snapshot driven development).

Meanwhile CPU power has also kept accelerating to the point that hermetic end to end tests that were unbearably slow can now be run on a laptop in seconds. Entire suites that used to take hours on CI can be trivially parallelized and run in minutes instead.

The tests you propose might take 10x longer to run. There is value in having quick tests as a first level of validation.

Mocks are tracking and storing behaviour and almost always are doing much more than fakes (which globalreset proposes) which are just simple (and ideally, correct) implementations.

> The point is: if your tests just assert which methods are called on dependencies with what arguments (or something close to that), they are extremely coupled and brittle.

I'll go further. These tests are worse than useless; they're harmful.

- They're the very definition of testing an implementation

- You will wasted a ton of time updating them when your implementation changes

- They give a false sense of security/productivity

Sometimes they're still necessary. E.g. many devices need particular sequencing in their power-up between multiple rails. The Intel 82598 Ethernet Controller datasheet[1] page 129-132 shows this nicely. If you're writing a driver you have to test that the appropriate GPIO control calls happen in the right sequence (and with the right timings). Of course it's an implementation detail, different ethernet controllers have different power-up sequence requirements, and thus different drivers.

Of course you don't want to have to use real hardware for these tests, especially because with some devices incorrect power-up sequencing can cause permanent damage. So you write mocks for the various hardware functions and test the driver with those, and use their counters & the test harness's logging to assert that the sequencing and timing is correct.

[1] https://www.intel.com/content/dam/www/public/us/en/documents...

There are entirely separate communities IMO when it comes to automated tests.

There are the pushing-the-edge-of-excellence folks, constantly refining methods and approaches, who tend to be passionate about the holistic benefits of testing.

And there are the folks who write convoluted tests that, when you dig into them, just confirm that String's .equals() works OK. DAO/database tests which have mocking going on to the point where you're faking a database response of "hello", say, and then checking "expected response == hello". Then yay our test passes!!

Personally I feel our energy should somehow be spent collectively on getting the latter group to just not write total garbage, and that the bar for "good enough" is really not that high, after which we get into diminishing-returns territory. And I think, personally, that mocking is responsible for so much confusion in folks that ultimately don't really know what they're trying to accomplish.

It's all about staying in language though, at least in my opinion. The test suite I don't run is the one that requires the complicated docker compose stack and the local network to look just right.

The test suite I do run is the one where I hit "run" in my IDE and then it lets be jump to what failed and debug seconds later.

That, IMO, is what mocking is about: making the tests fast and easy to run so they actually are.

Agree. Mocking is a slippery slope I think.

We need to mock, for example, DownstreamAPI01 that our app POSTS to and GETs from, so we can put in accurate and realistic faked responses.

We don't strictly need to mock other bits of our own apps - but many people choose to do so.

Martin Fowler writes very well about this, and I hadn't even appreciated the "split" between sociable unit tests and solitary tests:

https://martinfowler.com/bliki/UnitTest.html

I can see the appeal of both sides, but (a) I prefer sociable tests as they tend to test real code more than artificial constructs, and (b) people tend to get into a mess when they try to produce pure solitary tests, leading to the "not really testing your app" example in my earlier comment.

95% of unit tests will bring 5% of value and 5% of unit tests will bring 95% of value.

Most unit tests should be handled at the integration level.

The compose stack tests (when engineered right), are the only ones which can reliably reproduce and test almost any bug or feature reliably.

In 2018 I could understand the aversion to them given how unbearably slow and flaky they could be but those problems are draining away with improved tooling and faster machines.

I think before long they'll be the default - a test that is 4 seconds slower and 30% more realistic will seem like a no brainer.

I have both. 90% of my tests are normal unit tests that use mocks or fakes as needed to test the things I care about. 9% are integration type tests that validate broad behaviors with my app’s direct dependencies in an ephemeral environment managed via docker compose (usually anyway). 1% is a couple end to end tests that validate the whole system at deploy time and beyond.

I cannot count the number of python and perl tests leveraging mocks that we inherited that after we dug into them, there tests were asserting that 1=1 exactly as you say. "Make the db return hello, insure we returned hello; yay, framework works?".

Unit tests should validate error paths are properly exercised and that we handle expected results appropriately. It is wild how often this is not expressed in unit tests.

On the other hand, I've been surprised just how often a simple "1=1 in a roundabout way" sanity test explodes in my face. The code people write is just that shit. And so is, often, my understanding of it. So I always like to keep around tests confirming we can shuffle some data back and forth on the "golden path" - they're good documentation, good at detecting broken refactors, ad well... if you're going to point out that a test effectively checks if String.equals() works OK - sure, but that's an implementation detail, to which the test is oblivious, and you should be too.

Tests are often extremely helpful, but I think the goal of 100% coverage has done much more harm than good, and leads to the kind of trivial tests you describe.

TDD usually works well, and helps guide you towards designing a usable, testable API. Don't test getters and setters, test the actual logic you care about.

Such a bad take IMHO. Sorry.

1. You still need to update the fake object once you gonna add a new behaviour.

2. Fake object has a tendency to become logic heavy. Someone will add a stupid-not-needed-map to test some shit you don't need to test in this unit\layer.

3. You only need to mock the behaviour you depend on. If you've added new Method and you need to mock it despite the fact you're not using it in the code you're testing - its a bit weird and smelly. Consider rethink SOLID principles at least.

4. Go. Gen. Effectively you can automate mock generation. Debatable but it's cool.

Just. Keep. Things. Simple. Depend on what you need. Mock behaviour you need. Try to test flattened structure - do not test the layer below the one you're testing ATM (unit tests).

Mocks force you to be aware of how your method/function under testing operates at a low level. At that point your test is now tightly coupled to a specific implementation. YMMV, but I can safely say from working on large codebases across several companies I've seen this get painful at scale when things change, whether it's a simple refactoring or an optimization pass. Mocks also tend to be much, much more complicated on a per test basis.

All that said, I personally prefer the upfront investment in stubs. At scale, it is something readily reusable by other test suites and teams out there.

how come mocks make you tightly coupled with implementation?

Use and depend on abstractions maybe?

I once worked with a mockist who mocked out hashtable. It didn't end well when we had to debug his code.

I once knew a mockist who mocked out all the external code's behavior, but didn't add tests to ensure that the external code behaved as they expected.

That didn't end well either.

Just don't use mocks, unless they're the simplest thing that could work (usually not).

what does it mean “mocked out hashtable”?

Whole hashtable? Or what?

You mock interface/api of “something” you depend on to model the behavior you might deal with within the part of code you test.

thats it. nothing else.

during unit test phase I want to make sure that this exact code works as a state machine given all the possible inputs/outputs and that it handles any possible situation any dependency can cause.

If you need to debug code BECAUSE of the tests and not WITH the tests then something is wrong with the architecture or approach.

tldr. mock dependencies. model behavior of the dependencies. make sure you can debug the code with the help of your unit tests on the very layer you’re working with at the moment.

I find that mock-heavy tests I've interacted with are low ROI: they don't find or predict bugs, require significant work to write, and often have to be updated when the code under test has changed even when the actual feature/case being tested has not changed.

While mocks can be useful, IMO "never use mocks" isn't a terrible rule of thumb.

I feel that if most of your tests relies on mocks it is a symptom of a very side-effected programming style. If you cannot unit test most your business logic then you are probably mixing up your business level code with side-effecting code (like calling APIs or DBs).

Personally I've found it better to split pure business logic and side-effecting logic as close to the edge (API endpoint handler) as possible. Then it is trivial to see where side-effects happen and you get to easily write unit tests for the business logic that matter.

Mocking is mostly always a pain and I would rather not mock at all and do proper end-to-end for integration code if possible.

The 'business logic' for many apps consists entirely of calling other APIs and databases.

If your database isn't involved in your business logic then you're probably significantly underutilizing capabilities of modern databases.

How is that so?

For SQL databases at least:

You can your db sum columns faster than you can grab the data, parse it, and compute the sum. In query calculations to avoid race conditions from doing the math on separate servers, etc.

Triggers and procedures are a thing.

The nosql/kv store hype missed a lot of stuff relational/sql dbs did well. Mostly because at the time they declared sql was too hard, or just never studied anything.

> business level code with side-effecting code (like calling APIs or DBs).

Sometimes the business logic is in DBs with stored procs.

That’s a whole other can of worms! The difficulty of versioning stored procedures is such a detriment to effectively using all that power.

They should be part of your migrations? and your migration should be versioned?

The topic of software theory (articles, advices, recommendations) for testing is the biggest failure of this industry

Everyone is trying to figure out how to do it "right"(measuring by their needs)

and all of them struggle to realize that it is always context dependent - two different products, teams, companies may have different expectations and needs

I dont know how this happens that out of all arguable things in software engineering - is it that tests are the most chaotic ones, when up to the principle they are simple: if your code doesnt match specification, then scream!

Dont even get me on how TDD saves the world and is the only way how all software should be written (it is especially funny that tdd is accidentally successful by forcing api design first, yet ppl always argue for it due to red green transition and never due to api design first)

Also the concept of unit as in unit test may differ by kind of software

E.g unit test for parser may feel like e2e test for somebody who works in web apps

This topic of testing techniques is fraught with extreme points of view. The way I see it is that unit testing, int-testing, mocks, stubs, and now fakes are all techniques and tools in your toolbox. Do not throw out a tool in favor of always using one of them. Instead, I implore devs, to try and use the *best tool for the job at the time for a specific test. Understanding that "best" is based on opinions and current constraints at the time. I would implore exploration, and experiments to try these different tools to get a better understanding of their pro's and limitations.

Totally agree. In the past year, I've started getting my org into writing tests, and it's pretty hard to find the "right" way of doing things. Everyone is so opinionated, which is fine but they're all opinionated in different directions.

I've found some solace in finding one person that's really experienced in writing tests for my specific platform, and kind of following their blog as a "bible" of sorts. cough https://kentcdodds.com/blog cough

Nah I'm about done working on rails codebases with 2+ hour CI cycles because developers insist on hammering a database in unit tests.

There's value in having database tests. It lets you assert the logic embedded in SQL in repos, which you cannot test without actually running the query on an actual database.

Full green tests when app doesnt even wake up due to db issues sucks too

I can see a two-tier approach working. First run tests with mocks that runs very quickly, because no DB or API calls are made, they are mocked. Then run a full-on test that includes DB calls, because this will fully test your app better than mocks ever could.

This way if a mocked test fails you find out much faster.

This is my take. I have one test suite that should be able to run upon file-save and be completed in about 5 seconds. Then another suite that takes about 20 minutes that I can run in CI or locally. I suppose unit vs integration, but I don't worry about conforming to those definitions too strongly. Basically tests in the first suite should be fast and able to run in parallel. If they touch the file system or talk to a local db that's fine as long as it's fast and safe in parallel. The second suite can do whatever.

If you can use sqlite in tests it’s often a good middle ground

And I'm done with developers who proudly boast of 95% test coverage using mocked tests because "we don't want to test the database" and then have database changes cause production crashes. "But my tests were all green!"

Databases, external APIs, dependency version updates, etc are among the most likely places where unintended changes get introduced. Why would you not test that when catching unintended breaking changes is one of the primary purposes of tests? Your simple business logic layer function converting one object into another is not going to be the problem, the database is

Get the best of both world by mocking the database API itself. Use the mocks to memoize the db output. Disable the mocks if you need long thorough tests. Determine if the mocks need to be disabled by comparing the date of the last thorough check with the date of the last db schema change. Instead of something global like this, one could come up with something local to the test file, using a system of cassettes. Doing something like this would require to make the database a value and memoize according to its current state, but I think it's doable.

2+ hour CI means something else is going wrong. at shopify, our CI is like ~10 minutes and we typically avoid mocks.

That may be the wall clock time but how parallelized is that?

I'm not sure I follow the argument. Use fakes instead of mocks, but if you change the dep1 to have a method add instead of addone, you still need to update the fake. The effect is the same as if you needed to update the mock setup. About the only benefit I can see, is if the fake is used in multiple tests and you only need to update it once.

This misses the point. It briefly touches the point, then . . . facepalm.

If you test behaviors, either with mocks or with fakes, you're not tied to implementation details. I can only assume this stops things from being what the author calls "brittle."

If your test doesn't test a behavior, then you are testing (at best!) an implementation detail. So maybe remove the test. If you're looking at code coverage, you should be able to get complete coverage testing at this level without testing implementation details or . . . you've found dead code - remove that, too! Tell your boss what a wise greybeard you must be because your commits remove bad things.

Whether or not you test behaviors, there are more running processes with fakes. There's a lot more happening that can fail. I'd call that brittle. Fakes have their place, but the reason to use them is because they're much less brittle (prone to fail) than full-stack end-to-end tests are. You can spin up your new microservice with all of its shared-nothing data stores, surround it with fakes, and completely exercise all of its reachable code - code like those error paths that are hard to hit otherwise. Better to test as much as you can in isolation first - faster feedback, easier to diagnose failures, and less test maintenance than a deployed environment.

I mostly agree, but with one exception - I think mocks are still useful for RPCs to other systems, where you want to test that you:

1) Call the right method on the interface with the right data, and

2) Correctly handle different return values from the RPC.

If you change your code to call a different RPC method your test _should_ fail, I'd argue that's exactly the point of this sort of test.

On the other hand, for testing code that you entirely control, I 100% agree that a fake would provide a better test,

Where I have found mocking useful or unavoidable is when we need to modify behaviour that is not in our control. This almost always is in calling a third party resource (api or store) or an sdk or library.

Everything else can be controlled and tested via DI.

I agree with the premise and have taken a deeper dive (https://github.com/osmyn/Practices/blob/master/UnitTestGuide...) if anyone is interested in learning more about creating tests that aren't brittle. I did this after suffering through poorly written tests that made our team afraid of any refactoring.

My goto simple example here for dumb mocks is a multiply function.

You are doing it wrong if you have:

MyMock.Mocks(myMultiplyFunc).WithArgs(3, 4).ToCall(myAddFunc).Times(3).WithArg(4).ExpectsResponse(12).

Your multiply func being backed by add is not important to the unit test for multiplying. If you change it to have special handling for bitshifting in different cases, your mock tests break. Badly.

Mocks add coupling. Coupling is brittle. There are other testing options available. Are mocks sometimes useful? Maybe. I've not needed a mock library in 10 years of Go development designing and building robustly tested distributed sytems at scale. Every time I've seen a mock used it is gross. You end up with SomeSAASMock that is auto-generated to provide all the special handling of the real calls to the SAAS.

Don't mock out the full SendGrid API. Make a dependency interface "SendEmail(userID int, content []byte) (SendGridResp, error)". Inject that. Now you just have MyFakeSendGrid structs in tests and you can have it return an error when you want, or not. It is really that simple.

Depending on how detailed your integration or acceptance tests are, you can have sinks that capture network calls and return stubbed data or call the actual service.

> Make a dependency interface "SendEmail(userID int, content []byte) (SendGridResp, error)". Inject that.

Am I an idiot for thinking of this as a Mock? This is how I mock things anyway

It is maybe a semantics issue. See https://martinfowler.com/articles/mocksArentStubs.html.

Traditionally, a mock asserts against internal behavior while a fake does not.

I think it probably is, it seems like certain parts of the tech community has a more specific language for talking about things, and other parts have a less specific.

Mock vs a fake. The difference lies in where the behavior is specified. In a fake, the behavior is hardcoded or simple. With a mock, it is each test that says what the behavior is.

The two scale differently. Mocks are per test, fakes are once.

I think this one is context dependent.

If you are a small or medium project that doesn't make many RPCs just test with the real thing.

Once I got to google which makes many RPCs in every layer it just wasn't viable to not use mocking. I use mocks in almost every thing.

Yes, it makes unit tests brittle and couples to implementation. But unit tests are cheap and often need to be changed anyways. It also allows you to have control over certain situations like what happens if an error occurs. Which can be hard to test for in integration tests.

In short I feel like avoid mocks if you can but embrace them if you can't.

Do they still have guitar? I thought guitar was a really cool project

Yeah, still use guitar.

This article is extreme, in the case of Java (mockito), using mocks is a great way to infer the path the logic takes.

If anything mocking is useful for verifying that something is called whether that be a service or a repository method.

Equally mocking things like mappers is overkill, a nice balance is to inject some mocks and some "real" instances of dependencies into the class that is under test.

The added benefit is that you are testing the dependencies (if the mapper for example) are working.

Verifying that something is called is the canonical use case for a mock. If you're using a mock and it's not doing verification, it's not a mock. It's something else.

A very good way to use mocks (not like illustrated here) is explained in the GOOS book by Freeman and Pryce: http://www.growing-object-oriented-software.com/

Can you summarize what that good way look like for those of us that haven't read it ?

Does anyone know if there are libraries for writing fakes ? I wonder why there is a lot of mock libraries but I know none for writing fakes.

https://github.com/rr/rr

"Currently RR implements mocks, stubs, proxies, and spies. Fakes usually require custom code, so it is beyond the scope of RR."

Ah yes - I'm getting confused between fakes and stubs.

Either way, RR is good to have in the toolbox.

You know, faking stuff in code is a really case-by-case deal. You've got libraries out there like Localstack, for instance, that do a solid job at pretending to be AWS, but finding one solution that fits all? That's tricky, because everyone's needs are a bit different.

I've actually messed around with a library that makes stubs by recording external calls to things like APIs or databases. Sure, it can get a little brittle at times, but it's kind of cool because it's using real calls to create these stubs, and you don't have to put in a ton of extra work. Like anything else, it's about weighing the pros and cons and seeing what fits best for your project.

fff - https://github.com/meekrosoft/fff

Reply on news.ycombinator.com