Prefer Fakes over Mocks
tyrrrz.me
tyrrrz.me
The end goal is still avoiding defects, helping developers is just the "middle-man".
There’s always someone who’s willing to plow ahead, safeties off, and ignore the warning signs they see and the complaints of their peers. Denial is a hell of a drug (“That bug can’t be in my code, la la la can’t hear you look how fast I’m going!”).
Some people need a little back pressure.
Bugs can be hard to cause, and thus not be noticed by user acceptance testing. So for catching them, tests are more useful.
The simplest way of doing this is to write code with good practices then do a load of manual system testing to find bugs (system testing including user acceptance testing in this example). This is basically what most people were doing 20 years ago.
This can work, but doing the system testing is time consuming and so expensive and so makes making changes expensive. Instead you automate some or all of your system testing. This makes them cheaper, but these tests are expensive to maintain and still time consuming to run. If the test fails you don't have that much information about the problem.
To complement (or maybe replace) your automated system testing, you add tests for smaller and smaller chunks of software, allowing for tighter feedback loops and more precise diagnosis (there are also some negative tradeoffs). Applied well, this allows you to write features more quickly, refactor more confidently and generally develop more efficiently.
Basically, it's not so much about catching bugs as it is about getting to your desired level of reliability in the most cost and time efficient way.
This is like the physics joke about spherical cows.
You cannot assume this. Good software doesn’t just happen, it takes a ton of work, a lot of which is invisible - if you succeed. People might get lucky or cleverly hide problems in version 1 but in a long enough time horizon, the scales fall away and the hot messes generally reveal themselves.
Testing can be about a lot more than just catching bugs. It can be about any measure of reality that can be turned into useful feedback to make developers more aware.
Testing is like the headlights on your car - they're useful even in absence of bugs.
Also, in dynamic languages, you might even have tests that don’t break when you change (or sometimes even delete) the dependency because the mock was correct. That leads to a false sense of confidence.
Say I'm writing tests for lets say ShoppingCart.
ShoppingCart depends on an implementation of ICartDiscountCalculator (CartDiscountCalculator)
CartDiscountCalculator depends on a number of implementations of ICartDiscountStrategy (ChristmasCartDiscountStrategy, VIPCustomerDiscountStrategy, BulkPurchaseDiscountStrategy, etc)
Each of those, in turn has their own dependencies, perhaps webservice calls, db calls, accessing caches, etc.
The way I would test ShoppingCart would be to mock out ICartDiscountCalculator, and provide known responses.
Are you suggesting instead that you build up the dependency graph - i.e the actual implementation of the discount calculator, the strategies, etc - and only mock when it goes to external code (eg the DB calls)?
This would seem to make testing very very difficult - since each test needs to know implementation details of everything it depends on.
[edit] In other words, you don't test that "if discount is 10%, adding item to shopping cart increases cart total by 0.9×<price>"; you do test that "if user X adds item Y to shopping cart, and X has a 10% discount code, then shopping cart price increase by 0.9×<price>". This does seem like a headache, but it has benefits - in that it tests actual product scenarios, and you can now easily add variations like "if discount code only applies for books, and item I added isn't a book, then shopping cart total increases by <price>".
One of the problems I'm dealing with at the moment is exactly this problem - someone's gone and written a whole ton of tests which involve spinning up pretty much everything short of the DB, and now writing a small test that just tests the behaviour of one class is exceedingly difficult.
To take the shopping cart example I gave, if I write it the way that I understand you're suggesting, I would have to consider the behavior of all of the ICartDiscountStrategy implementations.
For instance, I've written tests for CartDiscountCalculator and ShoppingCart that have 10 different items in the cart from Acme Corp Manufacturing.
Some time later, I implement a new AcmeCorpDiscountStrategy which results in a different discount being returned. Now my tests for CartDiscountCalculator and ShoppingCart need to be modified.
I also need to potentially modify everything else up the dependency graph from CartDiscountCalculator - so my internal API that is used by customer service, an API used by mobiles, and the front end website.
Using Mocks/Fakes to isolate the ShoppingCart, it doesn't matter what the strategies are, we only care about how it interacts with ICartDiscountCalculator.
That interaction might change, maybe ShoppingCart uses the CartDiscountCalculator in a different way yeah, that'll break ShoppingCart Unit Tests - but it's that method only, and you're already in there modifying ShoppingCart anyway.
No, you would need to consider only the business requirements.
> Some time later, I implement a new AcmeCorpDiscountStrategy which results in a different discount being returned.
Why did you implement that? Is it a business requirement change, that "existing customers should get this different discount"? If yes, then your tests break - but for a good reason. If it's not about existing customers, but new sort of customers, then your tests should be unaffected and you just need to write new tests that target these new sorts of customers and validate that the new discount strategy is applied.
But, I'd suggest if you can't test a single class in isolation easily (and you might need fakes/mocks) then the class probably needs redesigning.
For your use case, if the shopping cart is simple enough, I'll just choose a simple or frequently used discount calculator and make some unit tests to test the shopping cart functionalities. Then for the rest of discount calculator cases there will be separated unit tests.
The problem with testing almost everything in isolation makes refactoring some small-inner components costly, since every change may break some of unit tests.
The question is why you want to test that one class? We do have some modules in our (elixir) system that are 'thoroughly' unit tested. These are modules that are algorithmically complicated so testing them in isolation is useful. Most of the code is tested in logical chunks though.
In unit specs you should use mocks/fakes for external behaviour. Then you can simply verify that it's calling the dependency correctly.
Isolation specs are useful because when integration tests fail, they usually give little feedback as to what exactly went wrong in the system.
Also integration specs are expensive and slow to run since they execute everything.
Finally I think mocks/fakes are useful because if you find yourself mocking 20 objects to get a single unit test working, then it's a sign that you need to redesign the code.
I would refer to these tests as "integration tests" although the language is poor here. Test a load of module together. This is beneficial in a number of ways: It tests whether modules are integrated together correctly - you can check that module A works and calls your fake of module B correctly, but you don't have a good way to check that your fake of module B matches module B behaviour properly (in practice it's usually when there are multiple modules involved that the problems occur). Unit tests only test stuff in isolation. It also means that you can refactor all that code, and as long as you don't change the top interface the tests will remain valid. If you move some responsibility around and change interfaces this can result in a lot of time consuming unit test changes, and that usually means that people don't actually structurally refactor as much as they should do.
Integration tests do not replace unit tests, they should just be taking a lot of the load. This at least is my interpretation of the well known "Write tests. Not too many. Mostly integration" tweet (https://twitter.com/rauchg/status/807626710350839808).
But that is an implementation detail. Only the BulkPurchaseDiscountStrategy knows whether it's 9 or 10 that's the trigger. Maybe it varies based on time of day or for certain customers.
> I would refer to these tests as "integration tests" although the language is poor here.
Agreed, it's a discussion I'm having at my job now, I havn't found particularly great wording. "Functional" testing still doesn't quite work right.
> Integration tests do not replace unit tests, they should just be taking a lot of the load.
Agreed on the first part. Integration tests are valuable, but I think they're more fragile.
Writing them is, IMO/IME, slower and more difficult.
As for the linked tweet, it's the "mostly integration" part that I disagree with.
Language is poor, but I think there is consensus on what "implementation detail" means, and I don't think that's it. Whether it's 9 or 10 is customer facing behaviour. It's behaviour that might come from your marketing team or product owner or whatever, but it is outward facing. It is something that someone could write in a word document and call part of your specification.
The implementation detail would be whether that logic lives in BulkPurchaseDiscountStrategy or as one case in a big ShoppingCart class. A user pressing buttons on your UI can see the result of the price break, but they can't see the difference in code organisation.
You can say "price breaks should occur at 9 items" without ever saying "BulkPurchaseDiscountStrategy". In one ideal, all tests would test user facing requirements and not know anything about your code. You could refactor your code to your heart's content and know you the result was still what you wanted. There are downsides to that though because those end to end or system tests are fragile to write and expensive to run. It's much easier to cover all possible code paths with more focused testing. Hence in this case you could test the top interface of ShoppingCart, with all it's assembled dependencies. Not the same as pushing button in a Web Browser, but a lot more convenient to write and run.
Point is that your implementation of the discounts is expected to change. I should not have to change my shopping cart tests at all just because the discount requirement changed - it makes the costs of those tests too high.
Of course this is domain specific. I expect discounts to change all the time. Likewise I expect taxes to change often. In both cases they need to be mocked/faked just to keep my test code sane. there are many other domain specific things that won't change. I'm always going to store the shipping mass in KG, and for some customers convert it to lbs. It is reasonable to hardcode the weight in lbs as that won't change. (future me who has to ship to mars will disagree, but I don't know what the requirements are)
* Change the (or shared test code) that has the price break specification from 10 to 11 * Run it, see the failures * Update the source code appropriately
The updated source code is the same in both our cases. In my case the changed test code is testing the ShoppingCart interface and in yours it's a unit test of something lower down. There's not extra cost, it's just in a different place.
Or are you suggesting that your tests don't know about 10 vs 11 at all? There's a reasonable architecture where such data comes from outside your system, and the spec for your system is 'price breaks should reflect this data input' and then you mock that data input and test it, but that's a different thing.
It might affect multiple tests, but all you should have to change is one or two places in some shared code pulled in by those tests. Too many tests have too much repeated code, and so become expensive to maintain, exactly like normal code. Refactor, think about shared functions etc. exactly as you would do with your production code.
Having multiple tests fail for the same reason is a waste of effort, either now or in the future (when new features are added or the constraints otherwise change).
Sometimes you have to depend on transitive properties to keep your test sizes down. If a function received the data, and the function has been tested thoroughly, why test it again while testing the callers?
Of course I try to make those fakes at as low a level as possible, but there's still a substantial part of my code that can't be unit tested, and needs test programs in the actual firmware to check it.
It's a fantastic comparison of the Detroit school of testing (what you and I believe) vs the London school, which is simply a different mental model about code and testing and mocks especially.
Framing the differences this way make it clearer why some people might disagree with us, and it's not because they're stupid or wrong- they just have a different school of thought on the matter. This doesn't excuse, however, the people who are doing a little bit of both and they don't know why, and they're getting the benefits of neither side of that coin.
For me, the biggest benefit is that I want to be able to refactor the internals of my code and have the tests (which map to business requirements, not implementation details) still pass.
It's also because a lot of the code I work with is inherently concurrent, and in my estimation, you're getting a lot more test coverage with the detroit model because a lot of your threading cases are simply covered if you use real objects and test it black-box style.
Because I like to test everything, not just little parts I isolate?
But if you read the article, you can see that this isn't the only school of thought.
Yes, integration tests are important too, but you'll want to minimize the number of scenarios you still need to test at that level.
There are a lot of really good "fakes" out there. Fake file system which lets you simulate disk failures. Fake time and randomness to introduce determinism. Every API over network can have a fake. These are all good things to fake out as you can simulate the appropriate failures which are almost impossible to do via integration, acceptance, or E2E tests.
The boundary between such fakes and mocks is obviously fuzzy: as we override more of a fake's behaviour we approach a mock, as we make a mock more functionally-complete we approach a fake.
Mocks (in this sense) are most useful when we want a lot of control over a very simple interaction; e.g. having Socket::read return a particular error state. We don't need a whole fake implementation for that, and indeed it's probably easier to just specify a mock with only those parts (and nothing else).
In my experience such situations are rare, since they tend to result from over-complication in the system under test, which makes it hard to test. Mocks are a useful crutch in those situations, but it's often better to refactor the code such that mocks aren't needed.
For example, whenever we create a mock that returns some canned response, that's a hint that our 'service object' is probably just a (possibly lazy) wrapper around that response. Any code that processes such responses can often be refactored into an ordinary function/method which accepts such response data as an argument. Sending test data into such functions is trivial (just call the function on the test data), and we can delete whatever wrapping/unwrapping boilerplate was previously littering the code.
When our mocks are more complicated that just spitting out canned responses (e.g. there's some temporal dependency between method calls, and correlations between their responses), those are exactly the things that fake implementations are for.
The lines have all been blurred and words have different meanings across different language ecosystems.
I summarize it in my own words below:
Dummy: Trivial implementation that might do nothing, return some default value, or throw exception if called. These are intended to be used only when an object gets passed but never used.
Fake: Have a working implementation, but takes some shortcut that makes them unsuitable for use in production. These shortcuts can potentially be pretty extreme, like a fake sales tax service that just uses a flat percentage for everything, an image watermarker that just returns the original image, etc. Basically if one could substitute the original implementation for this, still be spin up and use the whole app, without worrying about it crashing/misbehaving (such as hitting some unimplemented case or fixed return data causing map collisions etc), then it is a fake.
Stub: Harded implementation. Is not fully functional, and designed to have correct or usuable responses only for scenarios that the test will care about. (Sometimes designed to take into account a small group of tests).
Spy: A spy is an implementation that records some information about how it is used. It could be a stub or a fake. IT could also be a decorator patten wrapper around some other implementation. (Fowler incorrectly oversimplifies this by defining it to be a stub implementation). The recorded information can later be accessed to verify say the number of emails the code being tested tried to send.
Mock: An object that gets preprogrammed with expectations about how it will be called (and what it should return). Then after running the test, it is asked to confirm if the calls made upon it match the pre-programmed ones. One can pretty clearly see that this is a bit different than the others, in that it contains special verification logic.
-----
Either a spy or a mock is needed when the behavior being tested is not visible as either part of the return value, or as part of the state of some object after the test is completed. Those cases require "behavior verification", as opposed to "state verification". State verification might check some object's propeties, or query the fake database to see if the resulting state is as expected, while behavior verification.
The need for behavior verification can be common in certain "command" style apis. For example, if a dependency's purpose is to send emails, then it is quite possible that there is no state that remains (within the program) to verify if it got called. So the best that is possible is to verify that the dependency got called (or did not get called) as expected.
(Don't take the state vs behavior distinction too literally. Obviously both Mocks and Spies store recoded information as state. The important concept is that the normal implementations would not include this state.)
---
The article in question on which the above is all loosely based is: https://martinfowler.com/articles/mocksArentStubs.html
The good thing about (simple) fakes is, that you can see all the code in a very small class. No complicated library, that does something funky in a special case.
Mocking can be very useful though, but it can get too complicated very quickly.
But think about a blob storage class that has two methods, ReadOneFile(), ReadMultipleFiles().
You have one test that just mocks ReadOneFile(string fileName) and doesn't mock the ReadMultipleFiles(). You now change the implementation of the System Under Test (SUT) to use once the ReadMultipleFile(string[] fileNames) call instead of three times the ReadOneFile(). Now your test fails, but the implementation of the SUT is perfectly valid. You need to rewrite your test.
If you would use a fake, the test would stay green, without changing. The test with the fake is less specific to the implementation, and helps you better with refactoring/changing your code.
And additionally in this example ReadOneFile() is called three times, and is expected to return three different results. So even your mock needs some logic to handle that.
I guess my point is that if you have to agonise over mock vs fake then really it's a sign that your design is not quite as testable as you might like :)
If a mock object returning specific results from particular functions is enough to satisfy some code's requirements, then I would seriously consider whether that code could take those results as normal function arguments (possibly lazy/thunked).
Why would you not look into the fake implementations? If it's just a two liner that takes the parameters and writes them to disk, then that's a LGTM from me. You would not test every getter/setter method in any of your classes, would you?
If it's more complex than this, you can absolutely write tests for your fakes. In many cases fake data-stores use tested backends, and tested client libraries. If the client wrapper is complex enough then you can/should test it as well.
I'm not sure I buy this as something that's desirable to have to do, but it should address your concern.
These fakes are much less maintenance heavy. As changes to implementation details don't require changes to tests. Hence, whilst you need to write tests for your tests, you spend less time writing tests in total.
I argue that [0] the liability of mocks are much higher than the fake. With a fake, you could theoretically write tests that asserts that your class is satisfying the interface. There's no way to do this with a mock. You can just make an interface do anything, regardless of whether it makes sense. This is fragile under modification. With a large codebase, you're inevitably going to change the behavior of a class during a refactoring. Are you going to read every test that mocks that class to decide how the mocks should be rewritten? Probably not - the mocks force-overrode the behavior of the class, so your tests will still pass. Why would you look at them again?
> Make a fake for every error scenario?
By the time you're implementing a fake, you have an interface that the fake implements. One way that I've handled this is to make a FailingBlobStorage that takes an error in its constructor and throws it from any calls. If you need even more detailed behavior than that, you can create a mock for that specific test case. You're not married to the fake. It's not going to magically test every scenario. But fakes handle the common case in so many situations that it's actually surprising when you start trying them out.
[0] In fact, I've basically written this same blog post before. https://www.bitlog.com/2019/12/10/simple-software-engineerin...
Test for the relied upon behaviours.
It's essentially contract testing.
It's great objects that perform external actions.
But those are easy to find. This is why double-entry bookkeeping is the accounting standard: it's easy to add incorrectly once. It's much less common to add incorrectly the exact same way twice.
In some cases the fake is a lot more complex than the real implementation. (I worked on a diagnostic tool, the real implementation "just" needed to send/receive bytes, the fake needed to understand the bytes and return responses)
If I use Entity Framework Core I also love the in-memory provider (or SQLite in-memory), which gives you a very well faked database context for unit tests.
In some cases you can create end to end tests, that run and verify the complete business logic, without hitting the database at all. A mixture between unit and integration tests.
https://docs.microsoft.com/en-us/ef/core/miscellaneous/testi...
This approach easy to setup, however I found that it slows down unit tests especially with data migrations and setup before each test (to discard test data from other tests). I've since limited these to test only migrations and in some well thought out integration testing.
I can see fakes being useful in certain cases, but only just in certain cases.
Could be my inexperience with C#, but it didn't do justice describing the point in code samples.
My honest response is this: Writing tests should be quick and painless. Maintaining fakes is more painful than maintaining substitutes. Substitutes let me interrogate side effects in great detail. I’m not concerned will performance in unit tests. Ergo, substitutes are quicker and less painful, so IMO that’s a win.
But for real though, why should anyone care about someone else’s test architecture, of all things. As long as you neither under- or over-testing, why on earth does it matter?
If you have to maintain tests written by other devs you might start to care. Devs apply different patterns to writing tests and show different amounts of discipline doing so. Some produce lots of duplication ("it doesn't matter, it's just a unit test"), others prefer a complete setup and tear-down before and after every little assertion. I've seen things..
The example is a little contrived, but the whole article fits on one sheet of paper and does yield a better test than mocks would.
Testing against the real system is always better than testing against a fake. That should be your first preference, and if you can modify the system to make testing against it easier, that's even better. Sometimes you can't, and if your interaction is complicated and the parts that you care about are simple, a fake can get you good confidence that your part works. Mocks just feel like writing the code again -- in your code, you write "call a function with an argument, then call another function with an argument" and in the tests you write "pass if a function is called with an argument, and again with an argument". It can be the right thing at the right time, but it's usually writing a test to make some automated linter shut up. And that's a complete waste of time.
What do you mean here? Changing your testing system, or changing the real system?
This is why I'd want test to just have a lot more 'leeway' than normal code. Essentially I want heavy hooking, introspection and reflexivity but only for tests.
They do, yes. But:
* the mock syntax is complex and verbose. e.g. "repo.Setup(x => x.Method(It.IsAny<string>(), It.IsAny<List<int>>())) .ReturnsAsync(() => ...." is not the easiest syntax.
The syntax for a fake is just regular old "class FakeFoo: IFoo" with trivial method implementations and no library needed to make it work.
* the mock syntax does not scale. Your 5 lines of setup in one test does not help you in a different test class. So it gets repeated.
Soon you might find that the Fake is easier to read, easier to maintain, and fewer lines of code than all the mocks.
As the article says: "their self-contained nature makes them reusable as well, which lends itself to better long-term maintainability." Not all our code is "do it quickly and forget about it", especially where you see that same thing or minor variations of it done over and over again in related tests.
Mocks win here when you do something different, once only. Fakes win in the opposite cases.
> I can see fakes being useful in certain cases,
This is true, but it's also true that Fakes are _underused_ in general in in the .NET world. People reach for mocking frameworks as if they are the only tool.
A trivial mock implementation is less complex than what OP refers to as “fakes”.
One of the problems we get ourselves into is abstracting service io code into functions and calling them from our business logic code. This forces us to mock all the io calls, instead we should be doing the opposite i.e abstracting business logic into discrete functions of pure data in and out and then calling those functions from our io code. This way we can test the business logic independently of the io code. Testing the business logic now becomes quite simple and one can leverage property testing to further boost our confidence in such tests.
Many times the ramaining io code is just calling other functions and doesn't need to be tested.
Several other ideas that relate to this are hexagonal architecture, Domain Driver Design, clean architecture and functional core with an imperative shell.
Some resources to further explore these ideas:
Clean Architecture in Python - https://youtu.be/DJtef410XaM
Why integrated tests are a scam - https://youtu.be/VDfX44fZoMc
Boundaries - https://youtu.be/yTkzNHF6rMs
So in conclusion I would say that a lot of people use mocking frameworks incorrectly and the author is hitting home on that.
The core reason I find OOP to be superior for most classes of business application development in general—and web development in particular—is that external dependencies and other high cost interfaces can be delegated to their own method or instance variable then called. In general, I avoid inheritance if there is a way to approach a problem with composition, but testing via fakes are much more reliable to make by inheriting the original class and overriding a method or setting a default initialization parameter then calling the rest of the constructor. This takes the pain out of so many parts of testing that it's truly hard to understate.[0]
For example, say a library has a class that calls out to a third party service like S3 and an instance of that class is used in a business object in order to persist data. By introducing a fake that inherits from the library's class but overrides it's persistence method to effectively be a noop the rest of the validation, calculation, and other side effects that the instance does is conserved without the downside of waiting on a network call. It's not perfect, for example it could miss validation or side effects that only exist on the external service (e.g., S3 could change a mime type based on some crummy AWS logic) but it's a hell of a lot better than a mock that constantly lies to you about what its done. Furthermore, it's actually extremely simple to code up, unlike pure fakes that need to return hard coded values for certain method calls.
I know that something akin to this is theoretically possible in functional languages, but I've found that in practice it's more difficult. Though I appreciate other areas where functional languages shine, including situations with testing.
It's kinda hard to discuss because it really depends on so much context and good judgement and I've seen truly great development teams use languages like Scala in situations where I would have used Python with Numpy / Cython for speed. So don't take this as some hard and fast pronouncement. Just the general observations of someone that's been coding for decades and is reasonably well paid.
[0] I think the primary reason that I prefer integration testing over unit testing is that I've adapted to its shortcomings by the above approach.
> I know that something akin to this is theoretically possible in functional languages, but I've found that in practice it's more difficult.
Could you explain what you think is hard about it?
Higher-order functions: rather than taking a 'service object' as an argument, and calling one or two of its methods, we can accept the methods/functions we need as arguments. Tests can pass in their own functions, which looks a lot like mocking but doesn't need any tricks or frameworks. This is the easiest approach for simple cases, but incurs lots of boilerplate as it's used more and more.
Records: these are name/value collections (think '{foo: bar, ...}' in JS). Since the values can be functions, they're pretty much objects; but we can swap out these values without having to define classes, etc. We can use these just like 'service objects' in OOP.
Type classes/interfaces: these define certain operations/methods that can be performed on a given type, even if that type's generic. They're essentially records like above, but chosen by the compiler based on the type; some languages (e.g. Agda, Idris, Scala) also allow overrides to be taken from the current scope.
Domain-specific languages: we treat the methods/operations of our API as if they're builtin-functionality of some 'embedded' language. When we use such functionality, the result is a program for that embedded language. We can run these programs by defining an "interpreter" (a function which implements each of these operations); we can define a 'real' interpreter and a test interpreter. This is the most hardcore, and usually benefits from some framework for doing the necessary plumbing. I wouldn't recommend it as a first step, but I've had success doing this with Haskell's Polysemy library; another popular library is Haskell's 'Monad Transformer Library' but I've not used that myself.
I agree, tests should be simple! Dynamically generating stubs, mocks or even arguments for tests hides complexity which leads to test code that's hard to reason about... and is hard to change/fix
IMO, both mocks, fakes and "reals" (to put it in the same language) need to be first class citizens of software testing, each one of them used in the right circumstances. Just like resorting to unit testing only is as much a bad practice as it is to use acceptance testing only.
A good testing strategy should be adapted to each software and context, and potentially include all types of tests: unit, end-to-end, integration, numerical, acceptance, etc. and all sorts of resources: mocks, dummy data, substitutes, testing engineers, test databases, real databases, real users, etc.
The best solution doesn't need anything, e.g. if we have calls to our persistence layer mixed in with calculations, the latter shouldn't be tested with fakes/mocks; instead we should refactor the code so the calculations don't depend on the persistence layer.
If the behaviour of some code depends inextricably on some external system, then our tests should use that system. This avoids unnecessary code/work/duplication, allows tests to exercise more paths through our codebase, exposes problems in our understanding of that system, etc.
If calling an external system in our tests is dangerous, unacceptably slow, costly (if it's some Web service), etc. then we should make a fake implementation to test against. If our code only uses a small part of some large, complicated dependency, we might want to define an interface for the subset that we need, refactor our code to use that, and only have our fake implement that part.
If a fake implementation isn't practical, or would require so much overriding per-test as to be useless on its own, then I might consider using mocks. I would seriously consider whether that code could be refactored to avoid such coupling. I would also never make assertions about irrelevant details, like whether a particular method was called, or what order things are run in.
In this case instead of storing the blobs via a REST interface or a database connection, it just keeps it in memory. So it should mostly behave like the real implementation, but is not suitable for production, as it doesn't persist the data.
The only things that are missing there are real life latency or maybe exceptions because of connectivity issues.
So they are coupled to the implementation of the object in any case!
> It may also appear that the details we have to take into account here are not that different from the implementation-aware assumptions we were making when using mocks, as neither are actually governed by the interface of the component. However, the major distinction is that the coupling we institute here is between the test double and the real implementation of the component, rather than between the test double and the internal specifics of its consumer.
So shouldn't fakes have their own tests as well? Mocks don't need tests (at least in C# with a framework) because no new class is ever written.
Anyway most of these issues are inherently linked to the nature of OOP and are a direct trade off to its benefits.
Both Mocks and Fakes have advantages and drawbacks, but the OP doesn't reflect that.
I wish articles written on that subject would take a step back and explore how code can be written so that the burden of testing is reduced to a minimum.
In one case you create the class on your own, in the other case the mocking framework does that for you on the fly. With both approaches you can do very basic or very complicated things.
Or do you have a rule in your company that every class needs a test? Maybe this rule is the problem.
because no new class or method is created with mocks, it's merely declarative, no new logic is created than needs to be tested. Fakes very much have logic since they are implementations of classes.
> Or do you have a rule in your company that every class needs a test? Maybe this rule is the problem.
No Fakes are, not the fact that every unit should be tested. Fakes are very much a unit, the fact that they are classes is irrelevant.
furthermore:
https://news.ycombinator.com/item?id=24774752
which demonstrates my point. I don't want to have to write tests for tests.
On the best scenario, your mocks are just data that doesn't have timing characteristics, don't react to the system being tested, and are short enough to not need any kind of compression (in loops or gzip). On those cases, they don't need tests. But if you are doing anything weirder, they pretty much do.
I honestly don't see any sense in separating them in categories like if they had discreete non-overlapping mono-dimensional properties.
Anyway, why do you claim any of that is specific to OOP?
There is no need for Mocks or Fakes with pure functions.
Anyway, there isn't much difference between mocking a unity and passing some static data into your functions and comparing the results with the expected ones.
Instead of counting how many times a method is called (mock), your test just gets to test the outcome of the overall operation (fake).
That's an expectation which is an extra optional layer on top of mocking. The important difference between mock and fake is that a separate mock has to be created for every single test, but a fake needs only be created once for all tests that use the faked dependency.
I'd like to add my own thoughts to this discussion, but feel it's the kind of topic that blows up into a mess/mass of opinion such that adding another comment just adds to the dogpile; after a certain critical mass, no-one really reads the majority of comments before commenting themselves, and a positive feedback of repetition begins.
Maybe if the discussion was broken into narrow sub-topics this could be fixed?
https://docs.microsoft.com/en-us/visualstudio/test/isolating...
Which I initially thought was the topic being discussed.
So has Martin Fowler: https://martinfowler.com/articles/mocksArentStubs.html
The TL;DR: Mocks increase coupling and make systems harder to change. Fakes are more versatile.
What exactly is the difference between a Mock and a Fake? In one you store any side effecting values in memory with a predefined behavior in a separate class. In the other, you define the predefined behavior in the test class by intercepting calls to external components. Your test coupling to internal implementation is just as high, or higher. Your Fake needs to do exactly what you expect in isolation of the class that depends upon it, or the test result won't work. In addition, without the reflection capability provided by many mocking frameworks, it is impossible to "Fake" implementations of certain library components via `sealed` or `final` modifiers on external library classes. It is also impossible to verify side effects without a method to expose those side effect actions in your Fake, which will force, at the very least, an extended declaration in your test class.
Integration testing is an essential part of the test pyramid. When your required interface for your injected mock/fake changes, your test is going to have to change. Your tests will be just as coupled to your implementation with a fake. The coupling is just hidden in your fake implementation.
In addition, often we are mocking huge interfaces with complex inner machinery that we don't fully understand.
The real issue with mocking and faking is the necessity to mock and fake to isolate tests due to poor interface design in the class under test, usually focused around side-effecting methods.
All side effecting methods must return some sum type that is not void that indicates success or failure of the side-effecting actions. All side effecting actions taken by the code must be apparent at the type level of this success or failure wrapper. Tests must use random, generated input parameters to verify the returns generated are correct. And as always, a type signifies for all X, then Y; while a test signifies for this x in X, then this y in Y. Types are enforced by the compiler or runtime, which (though they may have a bugs themselves) are never wrong as the interpreters of code. What you get is what you get, even if it doesn't match up with what you think a particular type expression should output.
Even with effect encoding and property-based generated testing and compiler/runtime proofs that your program interfaces are correct, it is highly unlikely that a person incapable of designing and implementing a correct implementation would be capable of designing and implementing a correct test for that implementation or type hierarchy. Even with effect encoding at the type level all the way down, integration testing between components against live infrastructure remains the best way of detecting bugs -- all your unit testing and design assumptions are exercised by exhaustive business case testing.
Mathematical proofs derived from the type system are very strong validators of design. But they can only verify that the assumptions and constraints the designer has created are correct. Only verification via integration and user testing can verify that those assumptions and constraints actually meet the business requirements of a given design.
All of this is to say, if you are focusing on using Fakes over Mocks and relying upon tests to ensure correct behavior, eventually you will fail. In other words, bugs will eventually manifest in any system in which automated integration and user testing that verifies exhaustively the business cases requested by the business user is not part of the release process.
Unit tests and types are measures of the correctness of a given design with respect to itself. Unit tests and types are not the measure of the reliability of a given design with respect to the use of software artifacts, and cannot be. Nor can coupled/uncoupled unit tests improve the ease of maintenance of a program. Types and unit tests are a useful design and maintenance tools that can make the intent of a design clearer to the reader of a piece of code, and can help identify times while implementing where you are violating the constraints you have set for yourself. But they cannot make refactoring or interface changes easier to encode without inflicting line churn cost in the tests. If an injection is exposed to the user of a class, it is exposed, and you must deal with it.
First I'll introduce some axioms...
1. Mixing unit and integration testing is not ideal. If you are doing unit testing focus on verifying the code paths within the unit. When considering the collaborators focus on the finite number of states that are most likely to occur.
2. (I know this is contentions) Design your code so that it is easy to unit test. It'll be ok I promise. In almost all cases code designed for easy unit testing is synonymous with well engineered code
Imagine this python...
```
def get_a_file_and_do some_stuff(path):
d = dict()
with open(path, 'r') as file:
for line in file:
d[line] = d.get(line, 0) + 1
# a bunch of code that does something with this dictionary
return resultThe above is miserable to test with mocks. You end up trying to man handle the file object. Its not fun.
```
def get_a_file_and_do some_stuff(path):
d = convert_path_to_dictionary(path)
# a bunch of code that does something with this dictionary
return result
With the simple flick of the wrist I can now trivially mock out "convert_path_to_dictionary(path)" and I only ever have to work with dictionaries.
So you might say "but but what about the file dictionary code. are you going to test that?" Probably not and if experience is any indicator I'll never have an issue with it. The edge cases and regressions will all lie in the custom business logic executed on the dictionary.
I see engineers make their lives enormously difficult to live up to some unachievable standard. Often that standard yields very little value in excess of a much simpler approximation
I agree with you for most part. Mixing unit tests with integration tests is a recipe for failure.
"Make your code more testable by moving things that are cumbersome/expensive to test but extremely unlikely to break into functions you don't test. Instead focusing your efforts on making the items that are likely to break easy to test"
Phrased honestly it becomes more interesting discussion.
I have worked on projects that had static analysis and high code coverage standards where code quality was abysmal, the product was barely functional, and development was paralyzed. I have also worked on projects that had none of those things but managed to have happy customers and quality product.
What does this mean?
It means that writing lots of tests is not enough to guarantee good outcomes. 1 great test is an asset, 400 bad tests are a liability. What does guarantee good outcomes is focusing on "equity". Does writing this test improve my equity position? If the answer is no or if you can't qauntitatively demonstrate the value in terms of opportunity cost then the answer is obvious...don't write the test.
The current state of software quality at the development level closely parallels the plot of Moneyball. Consider this quote...
"Okay, people who run ball clubs, they think in terms of buying players. Your goal shouldn’t be to buy players. Your goal should be to buy wins and in order to buy wins, you need to buy runs. You’re trying to replace Johnny Damon. The Boston Red Sox see Johnny Damon and they see a star who’s worth seven and a half million dollars a year. When I see Johnny Damon, what I see is an imperfect understanding of where runs come from” ― Michael Lewis, Moneyball: The Art of Winning an Unfair Game
Many people try to buy quality with an imperfect understanding of where quality comes from.