Mocks Aren't Stubs (2007) | Hacker News Reader

Mocks Aren't Stubs (2007)

martinfowler.com

40 pointsbackslash_1631 comments

I feel like this distinction is arbitrary and unintuitive. A mock is literally just something that mimics something else. All "doubles" can be considered mocks. Claiming that mocks insist on call verification seems like a very forced categorization. If there's value in this distinction, maybe we need new terminology instead of asserting that everyone who's conflated "stubs" and "mocks" for two decades is wrong.

As a side note, verifying call patterns makes for terrible, brittle tests that cost far too much to maintain. This pattern is great for enforcing call patterns that truly that matter for, e.g, performance reasons (no unexpected cascade of DB calls), but is typically just a giant pain that provides little to no value, because instead of testing that the code does the right thing, you're testing that it doesn't change.

I think the difference is mocks carry tests while Stubs are dumb fakes.

I think the problem is the choice of words. Maybe mocks should actually be called "spies" instead and stubs "fakes".

That terminology makes much more sense, especially the "spies" part.

Personally, I tend to think of stubs as being super basic. A stub class for me returns default values or throws NotImplemented or something as simple as possible. It doesn't maintain meaningful state. Mocks for me are generally simplified implementations like an in-memory store that replaces a DB. Frameworks like Moq have muddied my definition though because they call things mocks when they only return canned data. Building a real mock (fake) with real state using Moq tends to be a pain.

Spies are a different thing: they allow any interaction, recording it all so that you can inspect it layer. Python's "mock" library, for example, is mostly a spy library.

> Python's "mock" library, for example, is mostly a spy library.

It's almost as if the experts don't all agree with this terminology.

Edit: This was a bit too snarky, but truthfully, I don't see evidence in the field that these definitions are widely used. I see evidence that an very small crowd is using these terms and asserting their correctness. Meanwhile the rest of the industry thinks anything they use as a "double" in a test is a "mock".

Even the people writing widely-used mocking libraries aren't using this terminology consistently.

There are objects that demand to be interacted with; there are objects that allow only certain predefined interactions; there are objects that allow any interaction and record them all for later inspection; and there are objects that are simple implementations of interfaces whose "real" implementation is more complex. Each of these object types exists in practice, and each is clearly an example of some larger category.

One group of people says "these should all be called mocks". Another group says "these are mocks, stubs, spies, and fakes, respectively, and collectively these are called test doubles". You can prefer the simplicity of calling them all "mocks" (simplicity is nice) but that leaves you with no way to distinguish between the types.

These categories seem like hair-splitting if you've written relatively few tests using test doubles. Once you've written a few thousand, you start to wish for even finer-grained distinctions. This is how basically everything works as you get deeper into the details: you need words for the fine-grained distinctions.

As far as I know, there has never been a competing set of terms. The only major positions are "we should use [the words I mentioned above]", which is mostly favored by those who have used a lot of doubles; and "we should not use separate words at all", mostly favored by those who haven't used a lot of doubles.

As a person who co-owns a company named Test Double, you'll never guess, dear reader, where I come down on this issue.

Here is a GitHub wiki where I break them down a decent amount: https://github.com/testdouble/contributing-tests/wiki/Test-D...

I've also written a JavaScript library called testdouble.js that has pretty detailed docs on their background purpose: https://github.com/testdouble/testdouble.js/blob/master/docs...

For what it's worth, Gerard Meszaros coined the term "test double" in his fantastic book (which most of you have never heard of, much less read) called xUnit Patterns. This is relevant to Gary's point, because like most things that require expertise specific terms only become useful as you grow close to the work. Most people don't use test doubles to facilitate isolated test-driven development, so most people won't come across the terms (or their sources).

I can't claim to be an expert in much, but—for reasons that evade my understanding—I do claim to be one in test double science™. What I can say is it can be tiresome when an expert has to constantly engage with relative novices who lack a curiosity with really engaging on the topic. Over the years, I've grown comfortable with the fact that most people—by which I mean, almost _all_ people—who want to talk about test doubles actually just want to fake something out in a test that's causing them pain. This is a perfectly fine thing to want to do, but it's orthogonal to why nuanced test double types & subtypes and jargon was ever necessary, which is mostly steeped in very rigorous schools of test-driven development.

There's no wrong way to do it. If you only write relatively "integrated" tests, nomenclature like "mock that out" won't cause you much harm but for the occasional eyeroll from people like me. But if you're interested in using tests as a sounding board to improve the design of your systems, they're one toolset for one style of approach (of many) that can help you in that endeavor—and which require the kind of specificity that Gary described above.

My feeling after reading your doc is that "mock" is an ill-defined term and asserting the specific meaning here is just prescriptive linguism.

Not only does your own doc agree that in common usage, "mock" is a generic term for any test double, your docs also assert that a "partial mock" is "any actual object which has been wrapped or changed to provide artificial responses to some methods but not others". So by their "precise" definitions, a "partial mock" is actually not a partial "mock". Come on.

One of the things that programmers do that I find most frustrating is proclaim that the definition of words that they use is the objectively correct definition, and frame their entire discussion as an argument that other people aren't using words correctly. Pick any term and the kind of argument comes to mind - "type," "object," "metaprogramming," "dependency injection," "design pattern," etc.

This article has a lot of good content about testing strategies, as well as an introduction to some of the jargon used in a certain testing discourse, but the argument is framed as telling you that mocks aren't stubs, and you, the reader, have been using these words incorrectly. But the reader has only been using these words differently.

This is a very common pattern of behavior among programmers and I think it is worthwhile to push back against it, separately from acknowledging the useful information in this article. (And the grandparent comment you replied to also did some of this as well, of course!)

I agree with you in general. In this case there's only one coherent taxonomy in widespread use so I think it's reasonable to say "just use it".

It's unfortunate that people who aren't already experts on a topic can't tell whether that kind of decree represents consensus or someone's idiosyncratic opinion (and there's plenty of both in general).

Framing is a different thing, though, and I can't comment there since I haven't re-read the whole article recently.

> One group of people says "these should all be called mocks". Another group says "these are mocks, stubs, spies, and fakes, respectively, and collectively these are called test doubles". You can prefer the simplicity of calling them all "mocks" (simplicity is nice) but that leaves you with no way to distinguish between the types.

I'm not saying that those should all be called mocks. I'm saying that are all called mocks by many developers already. There are frameworks that call themselves things like Moq and EasyMock that support multiple of the paradigms you describe. Picking "objects that demand to be interacted with" as the only real "mocks" feels arbitrary and frankly wrong given that the term "mock" has exactly nothing to do with requiring tests to cause certain method calls with certain params.

I'm totally fine with saying that we should recognize the difference between types of doubles and give them different names. I'm not on board with saying that suddenly a "method" is a function with no side effects as if that's some standard definition.

> and "we should not use separate words at all", mostly favored by those who haven't used a lot of doubles.

This is not subtle. And asserting that every with experience agrees with you does not make it true. I have written enough tests to have a legitimate opinion on this. I have also written test code filled with "mocks" that predates this article.

I didn't say that you are wrong. I didn't say "that everyone with experience agrees with [me]". I specifically used the words "mostly favored by" twice. Mostly does not mean "everyone".

It's fine if you want to use different words. But you either have words for the different types or not. And if you have separate words, you either use the five existing ones or try to push five new ones. I think it's significant that you objected to my claim about the popular taxonomy without suggesting five alternative terms with even a fraction of the adoption. (I've built an entire career partly on popularizing this stuff and I've never even encountered a competing set of terms!)

My comment at the bottom was pointing out that you basically called people who don't agree with you amateurs. Yes, you qualified it with "most" but the sentiment remains. This sort of response is especially unhelpful when the "amateurs" being dismissed are not laypeople but professionals in the same field. It's like a group of heart surgeons asserting that arteries are very specifically the blood vessels that carry blood from the heart to the lungs and then calling general practitioners inexperienced for not getting on board with the retroactive redefinition.

I also don't believe it's reasonable to assert that I have to provide new terminology in order to point out that this definition is fundamentally problematic and does not match the common usage. But if you want a word, call it a preset (because all the interactions are defined in advance) or a mine field (because if you step in the wrong place, it blows up your test, and because your grandkids will still be dealing with the mess).

It's possible to be a world-class programmer and think that test doubles are a silly waste of time. Many of the programmers whom I admire most think that. Nothing that I said precludes it. I now regret writing my monthly comment on the web.

I overreacted to a phrasing issue and it drowned out the rest of the discussion. That's my fault, and I'm sorry, because you legitimately tried to engage in discussion and I was rude. I appreciate your insight here and while I don't agree with the definition being asserted for "mock", I understand the desire for more fine-grained terminology.

I hope your next monthly comment isn't wasted on someone who overreacts to one tiny piece like I did.

I think you hit the nail on the head -- "as you get deeper into the details: you need words for the fine-grained distinctions."

Indeed, nomenclature gives us clarity of concept and that precision yields efficiency of thought and communication.

Are the names arbitrary? I certainly thought so when I started. How did mock objects became expectation testing instruments and stubs didn't? How exactly is a stubbed class not a fake? Or would we use the word fake only for something like an in memory database? Aren't all test doubles fakes? Except, if a spy just wraps the object class, but allows calls through, then is it a double at all, or would it just be a probe? So are partial mocks and proxies liars?

Regardless of whether the names are arbitrary, it's useful to have them. It sure beats "double type A" and "Type B double". "Perhaps you should use the Second Double Form, the Third Form isn't really necessary." Ugh.

Perhaps there isn't a competing name set because it's hard enough getting everyone to agree these are distinct concepts. Maybe that would be easier with better names. After all, if there's popular rejection of the distinctions until the underlying concepts are understood, then the names aren't leading us to water on their own merit.

It seems, also, the more I hear the generation before mine talk about doubles, the more I find they use them very sparingly. The Beck's and Fowler's seem laid back about the nomenclature in favor of shared processual understanding.

This happens all the time.

We have a vague notion of a thing, so we call it "X". Later, as we learn more, we realize that we can think of X as a category of things, and that one of those things, annoyingly, should probably be called "X". Now we have a problem. Some people will call the thing X and some people will continue to call the category X, because they read our blog posts from five years ago, because they were so popular, because we are so awesome.

This is a sign of a good thing, even though it causes confusion. Now we have to roam the countryside saying "I know we say 'X' to mean this general thing, but we also say 'X' to mean that more specific thing. I'm really sorry. It's my fault. I recommend that you say 'Y' to refer to the category, but be prepared to read 'X' on the web. It is what it is."

It is what it is.

Also most mocking libraries require that tested/mocked classes must be derived from an interface, when an interface is not required this adds overhead just for the purpose of testing, this is bad.

Correction: googletest/googlemock (in c++) can create mocks of non virtual classes, they use scripts to generate the mock code. Still you need a special test build to link with the mocked classes, in java/.net you dont have this problem because of reflection.

I often found that these terms are not very usefull when explaining the concepts of IoC in testing (which is essentially what they do) to colleagues who were unfamiliar with it. Programming in .NET I usually tried to use frameworks that avoided the terms, such as NSubstitute where you create an object by substitution:

var format = Substitute.For<IFormatCode>();

I haven't used it in a while but I remember that Rhino Mocks asked for ridiculous code like

var format = MockRepository.GenerateStub<IFormatCode>();

(Which one is it, a Mock or a Stub? And: Does it matter?)

I recommend Gerard Meszaros excellent book XUnit Test Patterns, he makes more fine grained distinctions between different aspects of testing. I liked his notion that you simply create objects that produce or react to "indirect input and output" (i.e. exceptions, opcodes, sideeffects) rather than "direct input and output" (which is what "arrange" and "assert" are checking). Mocks and stubs are only realizations of different patterns.

http://xunitpatterns.com/

That's an outstanding book. I'm not a Java programmer and that still might be the best book on testing, how to design tests, what to test and how think about testing at a high level that I've read.

Every validation you do on a mock is an assertion on outside behavior that is in no way coupled with your test. If that behavior changes, your test is now false. Not failing. False. That concerns me.

Yeah, maybe there is an approach that works well with mocks but this has always concerned me as well. It's rare (for me anyway) that I want to test which calls a function makes rather than if the function returns the correct result.

I can imagine scenarios where that is useful and I know how to use mocks when that comes up ... but the examples are always a system where mock based testing is strangely coupled to implementation, very fragile and have the scary failure mode you describe.

When you're testing functions in an impure language (pretty much any language, short of extremes like Elm, maybe Haskell, etc), functions are used not only for their return value but also their side effects.

Side effects can be things like making an http request or writing to a file, it can be changing some global variable, but it can also be something like calling a public method of another public object. Since that function/method could also be impure, the very act of calling it is actually, in a way, part of the contract of the function under test: it returns a value, and it creates a side effect in the form of calling another function.

The only real way to test for those side effects is to pass in a stub or mock and to check that the function was called correctly. That makes your tests brittle, but it's the only way to make sure it does what you think it does (even if you only call the function for an intermediate return value, the fact that changing the intermediate function could change the result of the function under test).

Those tests are brittle because functions with side effects are, in fact, brittle, and favoring pure functions (where the only thing that matter is the return value, and dependencies are composed or passed as arguments) makes both your tests and your code better.

Typically, I find that I'm concerned with what calls a function makes when those calls interface things external to my program. I.e. I'm asserting that a request to a given http endpoint is made, or that an smtp client sends an email. I assume the external libraries work and am testing my interface to them.

This is where I find myself having to use these. Though, I greatly prefer to have the test merely check for the result of the call, if I can. e.g., if I have to post to a url to delete something, check to see if said something exists. In the end, it is not the call that I care about, but what the call was supposed to do.

This way can lead to brittle tests, clearly. But the reality seems to be that brittle systems lead to brittle tests. There is no testing panacea.

Why would it be of concern? I think mocks have a good place in unit tests. I generally believe the premise of unit tests is to test your code against specific pre-conditions. If those pre-conditions cannot be created in a unit testing environment then mocks are justified. I would, however, be very way of mocks moving past unit testing and into integration or system testing (or other varieties like failure injection testing, performance testing, etc.). Mocks have their uses & their limitations. Understanding that is key to using them well.

> If those pre-conditions cannot be created in a unit testing environment

then you have technical debt to repay, by making those pre-conditions satisfiable

It means that you lose out on a lot of value from your test suite: besides verifying behavior on new code, tests also allow you to confidently refactor existing code. If your tests test implementation via mocking, and break when you refactor your code in ways that preserve the intended behaviors, then you're unable to confidently refactor.

Mocks lead you down a path where developers will eventually feel too scared or fatigued to refactor anything substantial.

This article is almost universally cited when a book or article on testing or TDD begins the inevitable discussion of mocks/fakes/stubs. But in real life, this is a distinction without difference. I have never once heard someone use the term 'mock' and be misunderstood because he/she should've used 'stub'.

How do you test your code that interacts with external components, like sending emails to a mail server or making an external API call without using some sort of test fake or stub? (using the definition of them from the article)

I'm new to TDD and refactoring legacy code for testability. With the code architecture I have it looks and feels like I need either a mock, fake, or stub.

I'm using an IExternalComponent interface as a parameter and for the concrete implementation I can either pass the real thing (it's non deterministic, this works if the service is up) or I can pass a test double that allows me to throw exceptions at will and force failure conditions to be tested.

Is there another way of doing this that I am missing?

Love this article.

Reply on news.ycombinator.com