Don't mock machine learning models in unit tests
eugeneyan.com
eugeneyan.com
Nerd pedantry: There's no "the".
https://en.wikipedia.org/wiki/Happy_Fun_Ball#/media/File:Hap...
Also, any test that calls out to another "function" (not necessarily a programming language function) is more than a unit test, and is usually considered an "integration" test (it tests that the code that calls out to something else is written properly).
In general, an integration point is sufficiently well covered if the logic for the integration is tested. If you properly apply DI (Dependency Inversion/Injection), replacing external function with a fake/mock/stub implementation allows the integration point to be sufficiently tested, depending on the quality of fake/mock/stub.
If you really want to test unpredictable output (this also applies to eg. performance testing), you want to introduce acceptable range (error deltas), and limit the test to exactly the point that's unpredictable by structuring the code appropriatelly. All the other code and tests should be able to trust that this bit of unpredictable behaviour is tested elsewhere and be able to test different outputs.
Fast tests are important. However I find that most integration tests run very fast if I select the right data. I can write unit tests that run very slot if I select the wrong data. I find that the local filesystem is very fast and so I don't worry about testing it (I place my data in a temporary directory - the problems with temporary directories are about security which I don't worry about in unit tests). Likewise I find that testing with a real database is fast enough (my DB is sqlite which supports in-memory databases - not everyone can work with a real database in that way). Which is to say you should challenge your assumptions about what makes a test slow and see if you can work around them.
But it's definitely not the only way to divide things up. The important thing is that the terminology your term uses is useful for your team.
One project I'm working on has two kinds of tests: unit tests run within the code and test classes, state transitions, etc. by calling the code directly, while integration tests are run in Selenium in a headless browser from the perspective of the user. Some of the unit tests on models do hit the DB (i.e. they do I/O).
The reality is that unit testing swept the world and has become synonymous with testing. The days of yore when things like capture and replay testing were it are behind us. Today, it is uncommon for anyone to write anything other than unit tests (as originally defined).
And that's why "unit testing" has become confusing. Its original meaning no longer serves a purpose. Today, we just call it "testing", leaving those who wish to continue to use "unit" for the sake of fashion to invent pet definitions; definitions not shared by anyone else.
We don't need a global definition of what a unit is, we just need to know what unit is being tested by a given unit test. This is important because it's how we understand coverage:
1. We need to understand what's within the unit (covered by the test) so we don't write redundant tests, and...
2. ...we need to understand what's not within the unit (not covered by the test) so we can write separate tests to cover that unit.
The unit can be anything, as long as we know what the unit we're testing when we read/write/run the test.
An integration test also needs to know what it's testing, but now you're looking at it from an outside perspective. You no longer care which specific areas of code (units) are within the integration, you just care what features are within the integration. Sometimes a refactor will mean that feature A no longer causes class B to call class C: that probably means changes to B's unit tests, because the code unit changed, but it doesn't mean changes to the integration test because the feature didn't change.
The important thing is to understand what's being tested by a given test.
For example, a project I'm working on has integration tests which are written with Selenium and run in a headless browser. Most of these tests involve users, but most of these tests aren't testing user creation, so I don't care how the users get created. Instead of going through the signup interface using Selenium, I'm just injecting the users into the database directly because that's faster and easier. The one exception is I'm not injecting the users into the database in my signup/login integration tests, because whether the user gets created is part of the feature under test.
Of course there are significant overlaps between these two categories, that is to say, there are some unit tests that are also integration tests.
Nothing says "doesn't bother me" like having to tell everyone you aren't bothered.
If there are no integrations, it's a unit test.
As in, it's all in the name. How does one fit a whole "service" (if we are talking something like a networking service like web API server there and not a DDD-style service) there really beats me, but I am happy to hear you out.
I'd note that things like setting up a client and then configuring it to access a service already fails that test, even if you hide that complexity into fixtures or "external" test setup.
Edit: for avoidance of any doubt, these are terms without clear, widely accepted definitions. I am mostly sharing how I am making what I think is a useful distinction to drive me to write properly scoped tests and structured code.
Two things of what? There are people that claim that as soon as there are more than one method involved, it's an integration test, and more common, that as soon as there are more than one class involved, it is. This lead to absurd mocking of utility classes to make sure that only one thing is tested at the time. And then, of course, the tests become very fragile.
And while there are situations when it is useful to only test one specific thing, what we really care about is the behaviour of whole system of many parts. We mock (fake or stub) to make the tests faster, repeatable and reliable. That is, what we don't want in our tests are slow calls, over slow networks or lengthy calculations. We want the calls to be repeatable, so we don't want to call, say, a database where the state changes between test runs. And we don't want to make calls to unreliable services; unreliable because they are under development or because they are over an unreliable network.
But if you need to use mocking there heavily, your code is not structured well: look for functional style and acyclic directed graph structure of calls.
Fashion, but there is no good reason for it to be there. It does not communicate anything.
There is a large number of people using terms like unit, integration, system or end-to-end to differentiate between different types of automated tests ran in an xUnit-style framework: why insist on not allowing the term to gain a new meaning?
Having 50k tests complete in that time when backed by memory on a modern CPU is trivial (easily achievable with proper unit and integration tests).
I've been on one too many projects where tests were exactly slow because developers didn't care about testing just-enough when unit- or integration-testing — not to mention becoming parallyzed by any tiny change breaking a ton of unrelated tests.
What you are talking about are tests that cover even more than just integration points, and are usually described as end-to-end or system tests (depending on the scope and context of the code under test). I suggest having as few of those, but in general people structure their code badly so sometimes it's hard to avoid it (most web frameworks don't help, for instance).
There is more than one way to solve the problem you suggest one alternative. I have found other answers that let me not need to worry so much.
When I'm deciding what to mock, I don't use "code under my control" as the determining factor. For me the question is, "is this the code I'm trying to test with this test?". If not, it's often a good idea to mock it even if it's your code, because you want your test failures to indicate where the error is.
Of course, you probably shouldn't be testing code that isn't yours, so that does mean you can mock code that isn't under your control.
For example, let's say you have two components, A and B, where tests of A end up triggering calls to B. In an ideal world, you have tests of B in isolation, so when writing tests of A, you don't need to test B again. If you don't mock B when you test A, then if something fails in B, you get failures in both A's tests and B's tests, and that makes it harder to figure out what's wrong. Instead, if you mock out B in A's tests, when something fails in B, you only get failures in B, and that pinpoints where the issue is.
There are caveats galore. The first caveat that comes to mind: If you're on a project with poor test coverage, B might not have any tests, and it may be worthwhile to write tests against A that don't mock B, so that you get some testing of B "for free". You can mock out B in A's tests once you have tests against B, but that may not be worth the effort. Yes, that's sort of blurring the lines between "unit test" and "integration test", but whether or not the code is tested at all is far more important than whether it's unit tested or integration tested.
[...]
LLMs are a prime candidate for mocking, because they're non-deterministic. Non-deterministic code it terrible for unit tests, because it will fail randomly, and over time that slowly normalizes ignoring test failures. You can make probablistic assertions on non-deterministic code, but there are two problems with this:
1. The more "wiggle room" you allow in your probablistic assertions, the less useful the assertion is--3rd-standard-deviation results are sometimes bugs, not just unusual results, and you don't want to ignore them, and...
2. Probablistic never equals deterministic; if you're running your tests often (and you should) then you'll get test failures on probablistic tests eventually. Broadening probablistic assertions only decreases the chance of failure, it doesn't remove it.
Probablistic tests create an impossible problem: if you tighten your assertions you struggle with meaningless failing tests and normalize ignoring test failures, while if you loosen your assertions your tests aren't really asserting anything any more. And there isn't a happy balance: if you go somewhere in the middle, you end up having both problems.
Mocking the LLM removes the non-determinism problem when the LLM is not the unit under test. That is to say, if you're testing code that calls the LLM, it may be a good idea to mock the LLM in those tests. Obviously, don't mock the LLM if your intent is to test the LLM.
Another possibility for removing non-determinism, which is probably more appropriate when you are testing the LLM, is to make it non-deterministic. You can do this by abstracting the RNG out of the LLM. Then when you want to test the LLM, you pass in an RNG that is seeded with a constant or even mock the RNG (gasp! You can do that!!?).
There are no silver bullets, and this approach has its downsides, because non-breaking changes to the LLM will likely result in changes to the now-deterministic outputs, meaning that a non-breaking change to the LLM code will still result in a bunch of breaking tests. This is why I would prefer mocking the LLM for tests that aren't intended for testing the LLM.
This problem is exacerbated if B is a popular object used by many components.
IMO if you own A and B, never use mock. Possibly write a Fake B if B is non deterministic or slow. Then write a parity test for B and Fake B
> This problem is exacerbated if B is a popular object used by many components.
This is a great addition to the caveats I mentioned.
> IMO if you own A and B, never use mock.
I'm going to take issue with your use of the word "never" here.
If you said something like, "If you own A and B, don't use mocks the vast majority of the time", I'd even agree--but there are important exceptions (and I'd argue that non-deterministic B is one such exception).
> Possibly write a Fake B if B is non deterministic or slow. Then write a parity test for B and Fake B
Would you mind explaining what you mean here?
Perfectly fair. I shouldn’t have stated it as an absolute. If I could still edit it, I would change my statement to “don’t default to a mock”
> Would you mind explaining what you mean here
Gladly. Martin Fowler provides really clear definitions for the different types of test doubles.
https://martinfowler.com/articles/mocksArentStubs.html
Maintaining that your Fakes are correct takes work. An easy way that I’ve found to do that is to run the tests against a “real” component and the Fake component with the exact same set of assertions and set up. If that test breaks, then you know that consuming code should also break
I haven't used fakes so take these critiques with a grain of salt, but I have two concerns with fakes:
1. The process you're describing to verify fakes does take work, and sounds suspiciously like it veers into testing your tests. That's probably worth it for applications like the space shuttle or an X-ray machine that need an extreme degree of reliability, but seems pretty overkill for the kinds of applications that make up most of the work in the software industry.
2. More often than not, I can't think of a useful fake for most things I'd use mocks for. The in-memory database mentioned by Fowler is an exception: you're skipping writing to disk, which saves a lot of time. But in most cases, the fastest implementation of a unit is... the unit, because if it wasn't the fastest you'd use the fake instead of the unit.
And all this sort of sets aside the larger issue which is that there are plenty of cases where the unit you want to test isn't the interface between A and B, so having a real B in that test just adds unwanted dependencies. The ideal here is that if you break the interface between A and B, one test fails--the test of that feature of the interface between A and B--which tells you exactly where the bug is. With an LLM, for example, this could look like you calling the real LLM and just testing that the call doesn't call any errors and returns syntactically valid output (even if the output is unpredictable, you can verify the syntax). If you're using fakes everywhere, then a failure of the interface between A and B is going to cause failures in a bunch of A tests, which makes debugging harder, not easier.
The case of the in-memory database makes sense because a general purpose fake (the in-memory database) can be written by a third-party and act as an effective mock for all the places you're using the DB--it's easier to use the in-memory database than to mock all the places where you call the DB (and notably, I've never seen anyone write a bunch of tests to test the in-memory database as you describe being a good idea for fakes). And in most cases, I'm using mocks because it's not easier to use the real thing.
Eg a simple example is having FakeSentimentAnalysis that will behave exactly like SentimentAnalysis for the classes it can return. You can then have an identical (parametrized) test that works on both, and from there on, you can trust that this test will break when fake diverges from the real implementation, without having to worry about mocks littered through code.
I like to push that even further and implement fakes inside the code/library/service that implements the real thing too: this ensures interfaces are really identical, and you only need to structure the code using the same approach.
As this is all code under your control, that is an ideal which is not hard to achieve if the entire team accepts that mindset.
I generally do that by having a test that works both against a fake and against the actual implementation, a bunch of others that only use fakes, and a few system/e2e tests covering the whole thing.
With not much effort, you get increased trust in the code you write.
But most notably, it makes you write testable code which I think is most maintainable and most readable code to write.
There is no such assumption. The assumption is that if you rely on a fake to service all your testing needs it cannot be tested against the real implementation as, in many cases, you do not want it to work the same way.
I don't know if a network connection failure is the greatest example, but let's run with it. Why bother adding simulated network failure into your fake, which, due to the problems you point out, won't be touched by your double-duty test suite anyway, when you can just create an additional mock that does nothing but simulate network failure? Why add needless complexity to the fake? You haven't gained a testability advantage.
That's not to say that a fake isn't also useful, but you haven't made a case for why it has to be the be all and end all. You can use both.
And sure, this does not gain any testability advantage compared to a mock, but if your test for the failure uses as much as possible of the same code paths as the real implementation, only substituting the fake in, you increase trustworthiness — if the APIs between a fake and a real implementation diverge (a common problem with mocks as tests continue to pass), it's likely to be caught by the double-duty test, and as you adapt your fake to match the new reality, you'll likely start getting the network-failure test to fail too.
In the above example, the only bits you can't fully trust, since it's not automatically tested for both implementations, is your "emulation" of the failure: you want to be careful about how you implement that so it really happens in comparable circumstances to the real implementation (eg. it's ok to throw an error where you would otherwise be calling out over network and returning data).
A lot of it depends on how you structure the code. In memory database fakes are the easiest examples, because it's clear to most people how you can structure the code to have a facade API that's used everywhere, and only have the final fake/real implementations that either do stuff in-memory or on the actual database. But you can generally do that with anything.
In general, testing is never equivalent to proving code works correctly, but I think this is the closest you can get (with a healthy dose of fuzzing on top).
However I found most software engineers not to believe it to be possible or doable with not much effort to achieve this level of trust in the code. But "showing the code" has managed to convince most — it does require a switch in the mindset, but it's quite similar to accepting that real TDD is possible for anything but toy problems (I don't think it's the most efficient way to develop, but I think it is possible and teaches people to write testable code).
> Don’t test external libraries. We can assume that external libraries work. Thus, no need to test data loaders, tokenizers, optimizers, etc.
I disagree with this. At $work I don't have all day to write perfect code. Neither does anyone else. I don't mock/substitute http anymore. I directly call my dependencies. If they fail, I try things out manually. If something goes wrong, I send them a message or go through their code if necessary.
Life is too short to be dogmatic about tests. Do what works for your (dysfunctional) organization.
Im mystified that they became the "default" test. They ought to be niche.
If you are making a standard library then you should be writing a lot of unit tests. However odds are that isn't your job.
I think code units / syntactic units became popular as a simplification of the original idea, people copying the style from examples and seeing it as a solution to their problems without actually understanding what the example was describing. Then it gets repeated over and over as "test individual functions" until the original idea is lost and most people haven't even heard of it.
Nowadays the original idea is partially described by black box testing.
The concepts the author is looking for are integration tests and release evals.
If you're just using a ready-made model in an app, for unit tests mocking the model out is fine. For integration tests, you probably want to use the actual model, of course (maybe with low temperature, to reduce flakiness?).
Machine Learning : Input Data + Expected Output = Learned Logic
Let me stop you right there. No logic is learned in this process.
[edit] Also, the LLM is inductive, not deductive. That is, it can only generalize based on observable facts, not universalize based on logical conditions. This also goes to the question of whether a logical statement itself can ever be arrived at by induction, such as whether the absence of life in the observable universe is a problem of our ability to observe or a generally applicable phenomenon. But for the purpose of LLMs we have to conclude that no, it can't find logic by reducing a set of outcomes, regardless of the size of the set. All it can do is find a set of incomprehensible equations that seem to fit the set in every example you throw at it. That's not logic, it's a lens.
They can explain addition to you but they can't perform it without calling out to an external API.
In dealing with complicated code, this is a very insidious problem, because they can often write functions that compile and run, but which miss edge cases because there is no actual logic behind them.
[edit] I should add that a sufficiently large NN may be able to create an entire computer inside itself that can run code, but even that would not be reliable because it will have arrived at the gating by inference rather than starting from a deterministic root.
Best evidence for this is the construction of natural numbers in traditional math* with `successor` function (A. There is a one (1). B. There exists a successor of one (succ(1)) called 2.) — everything else follows from that using formal logic, and that's why everything else is generally out of reach for generative ML models (eg. summation can be defined in multiple ways, one of them being successive application of a successor function the right number of times on one of the operands).
* Modern approach is to use set cardinality instead, but the essence is the same: something has to be memorized (axioms), and then logic is applied.
Take a 3-year-old who's holding two pieces of candy. Say, "I'm going to take these and give you two pieces back." Then give him one piece back. He's going to start crying. Perhaps he can't explain why, exactly. But if you ask him why it's not fair, he's going to tell you that two is more than one. Ask why is two more than one: It doesn't require succession. It's a fact of the reality we inhabit. We can measure it with our hands, our eyes, its weight. We can hear two sounds, two voices at once and differentiate them (non-blocking, nonlinear processing). AIs don't inhabit any reality grounded in the physical constraints they purport to understand, therefore they can never arrive at the conclusion that 2 > 1. They just appear to "know" it because it's part of the data set.
But once you understand that 2 > 1, then 3 must be > 2. If acquisition is hardwired into our brains, then so is the capacity for logically explaining why Y > X.
Food for thought- would a savant use the same process? Or have they somehow recruited more of their brain to where they can memorize much larger problems.
I think savants usually solve problems in the most deductive way, using reasoning that leads to other reasoning. I went to an elementary school in the 80s where more than half the kids would now be labeled autistic... some got into math programs at colleges by the age of 12. I believe it's all pure reasoning, not like some magical calculator that spat out answers they didn't understand the reasons for.
[edit] If you meant: Do savants solve problems by recursively breaking problems into smaller and smaller problems, then yes. But the breaking-apart-of-problems is actually the hard problem, not the solving.
https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mec...
>But for the purpose of LLMs we have to conclude that no, it can't find logic by reducing a set of outcomes, regardless of the size of the set.
A ML model is probably not going to converge strictly on a formal logic system practically and there's also the question of if formal logic even underpins the result of the dataset you're feeding it, but that's entirely different from saying it cannot in principle.
How does it matter whether I inline my test data inside the unit test code, or have my unit test code load that same data from a checked-in file instead?
As a rule of thumb, I also only assert on input/output values that are explicitly defined as part of the test body. Saves a ton of time chasing down fixture definitions.
- Algebraic Effects will land in mainstream languages, in the same way that anonymous lambda functions have
- This will render "mocks" pointless
Sorry… with a title like that, I couldn’t help myself.
"If you're ever lost in a wilderness setting, far from civilization, and need to be rescued, just start talking about unit testing. Somebody will immediately show up to tell you that you're doing it wrong."
This is just nonsense. It'd effectively mean you only had integration tests. While they are absolutely fantastic they are too slow during development.