Test Smarter, Not Harder: Focus on Outcomes, Not Outputs
markus.oberlehner.net
markus.oberlehner.net
The purpose of unit testing is to isolate the various scenarios by making sure that each unit/component/whatever correctly responds to its various inputs. In this way, we only need to set up the relevant scenario for that unit. If we tested each of these piece of functionality in an E2E test, then each test would also need to set up the user session, settings, etc. It would be a nightmare to author and maintain, compared to the simple, straight forward unit tests.
In contrast, E2E tests are good at making sure that the whole system works well together, and that each unit is getting inputs and providing outputs, and that the page doesn't break in the process. Both are needed to maintain confidence, and each solves a different problem.
If your E2E test is flaky, then how is your production environment reliable?
> they can require a lot of setup
Setup vs mock.
> they can be very slow, etc.
We have a good handle on scaling, slow is now a product of the testing time being smeared across your org like peanut butter.
> The purpose of unit testing...
Does E2E replace unit testing, no. Things that are already units or esoteric (business "logic", you know it's in your code base). But if your writing mocks your better off un fucking your environment that E2E testing is viable, and make it a first class citizen. Everything is a crud app of some sort: web, api, mobile, smoke signal... Dropping data in these front ends, can be validated out of the back end, and in the app itself.
To put a fine point on it: Unit tests make me feel good about checking in. End to end tests make me confident to deploy!
IME, E2E tests are flaky despite the underlying system not being flaky because E2E has to do a series of things, often including waiting/polling, and in real life people will wait longer or compensate but in tests you want to do it all perfectly and fast.
From an end-user's perspective, there's very little difference between "the task didn't work" and "the task worked, but took longer than your patience to wait." Both appear to be broken.
The flakiness comes from needing an automated browser interaction acting against a page with changing input. If your page is static, sure, the E2E test is probably straightforward and quick. But the moment any dynamic functionality is introduced to the page, the test's interactions need to account for the page's current context (or wait for an intended context). And that has to be coordinated amongst the actual test code, the testing browser, the driver, etc. If anyone one of them hiccups, you may end with a false negative, and those happen more often than we'd like.
This isn't an an end to end testing problem, this is a testing problem. Mocking isn't a fix for this (it's not a network issue). And testing in another environment (node/bun as units) is akin to deploying a server to linux boxes after having tested on windows (and I would argue far worse because the whole purpose of the front end is the dam UI).
Browser test: front end -> api -> db, full transaction life cycle inter coverage of the whole interface (crud operations).
Mobile: same as browser...
API tests: test client -> api -> db, full transaction lifecycle, should give coverage to the backend. It isn't the job of a browser test to hit every variant of every endpoint however, they should all be exercised from the front end.
You should be calling out to as many actual running services durning E2E testing, the system should behave in this environment the same way it will in production. NO mocks no synthetics, "full" data paths.
Unit testing:
Validation screams for "unit test". Does your "email" field allow for + encoding? Do you have frontend CC validation (does the card number hash) vs back end where it "auths" the card (auth is outside the scope of a unit). None of these sorts of test should need a mock. We're not faking another part of our stack (those should be end to end tests). Does that mean you should NEVE mock, and have "full" unit tests, no. But they should be sparse, for those things that have to be hard to understand or are complex (business "logic").
The bonus is you also have a wet blanket for scaling issues (if you do it right). Need to refactor or re-platform a service/endpoint? The tests are agnostic from the code, and you should have a "thin" set of unit tests to port!
I disagree. Assume it usually takes 1sec but 1% of the time it takes 120sec. Does your test suite wait 120sec and fail 1% of the time or does it wait 1sec and speed up the test suite? These are test trade-offs IMHO
Whether you use playwright or cypress getting the correct wait events figured out is a huge pain.
Only if you run pure server side applications that’s simple.
End to End testing is less about "making the front end do all the work" and more about being free of mocks and synthetics.
Just because the front end exercises my server, does not mean server devs get a pass on testing API down. The tests I write should inform the data your sending and receiving, they should function as an example of inputs and outputs of what to expect in the system. Those timing issues are either a real front end problem, or a server problem (and my own tests should have noted that I did not meet the SLA).
That really took the value of E2E down several notches in my estimation.
Same for any testing. it should be part of the cost and Cost Benefit Analysis.
Right now I'm still trying to convince people that fakes and mocks are not the same thing.
A good end to end test suite will give you a much truer indication of the state of your systems.
Of course I don’t say do zero unit tests, they have value, but I think too many orgs are too zealous about unit tests and too soft on higher level testing.
It’s apparent that the author has never worked on a legacy product.
End-to-end testing works great until your project grows in complexity and something doesn’t work right. Then you play the game of looking for where and why while your boss is frustratingly asking you how long it’s going to take while telling all the other bosses that you’re incompetent. Worst case, you break something, and you don’t realize it, or your coworkers break something you wrote and they don’t realize it.
Unit tests can protect you from those situations.
The author misses that component tests don’t have to be like unit tests at all, they are more like integration tests if you test entire pages (which are components too), use actual user interactions and mock only the I/O (mostly network) layer. It’s like E2E without spinning up a backend.
The main objection heard will be: "It takes x hours and n servers to set up an E2E environment."
Infrastructure is hard, and with cloud functionally unlimited, so people write a mock and move on with their life.
To do E2E well your culture needs to support it. You need to be able to spin up, run and summarize an E2E test in less than 20 minutes. Set your test up to END on the hour (not start, end) SO that if there is an outage no one can make the excuse that the "meeting already started". If the test fails every one who has changes in that build drops what they are doing and treats it like a production outage. The technological magpies, collecting shiny things and adding them to the stack has to stop, because you dont want to add "new tech" to build time. You may need to run some of your own infra to get this done. Some servers in a closet some where will do wonders for your bills.
Because you're always building your environment from scratch, in the event of a real disaster you know what it takes to put a large chunk of your environment back together. Your like going to have a pretty lean stack, one that scales diagonally (at this point go look at AWS spot pricing and how much you can save if you can scale that way).
That isn't to say you should not unit test. Things that are clearly stand alone, like well defined validation, things that are ugly (that 20 condition switch statement that powers some esoteric edge case of business "logic") should get unit tests. Places where you can leverage fuzzing easily should have unit tests! Anything that has a dependency tree or needs a mock, should be skipped and addressed with an end to end test.
What will you get in the end? A production environment with every bit of unneeded complexity stripped from it (and probably a lower cloud bill). An aversion to adding anything that doesn't have VALUE to your stack. A ruthless eye for what qualifies as "bloat" (hint, a LOT of it is bloat). The removal of code for codes sake (mocks) and layers of duplication (an E2E create user test can exercise a lot of code, and replace a lot of unit tests at various levels). The whole team having a much better understanding of how your infrastructure is built and deployed.
A few years into that job, gradual typing came to Python and I realized that a substantial fraction of the unit tests we were writing were in fact just making up for a lack of static analysis in Python. Since then I've come to believe that with decent typing and static analysis, unit tests are most useful during development of new code and only marginally useful after that. Integration tests are much harder to set up initially but are much more valuable.
Second: the rise of pedantic has given the Python ecosystem a great way to enforce typing and make up for python’s lack of SA. It can be really helpful to document what types are fed to which layers of the system. Since Pydantic will aggressively explode if users try to start throwing arbitrary models/data packaged around, it can really help if you’re working in a system where a model needs to change in one place rather than 10, and can lead to better hygiene than in the old days, where it was more common to pass less strict data wrappers around or have many layers of software defining their own interfaces rather than exporting that to a model which can be reused globally. In theory this is also possible with POPOs but in reality, duck-typing and mocks can lead developers into some really bad places where they think they’re typing things correctly until they get to the integration tests, which catch the bug and throw any unit work out the window for another iteration.
Great. Go over in that corner and argue with the other testers and some managers about defining "better" while the rest of us get some work done.
> strong emphasis on unit tests, and while they have their place, let’s not forget about a category of tests that I consider much more valuable when it comes to testing complex web application user interfaces: End-to-End
First, testing UI doesn't mean the tests have to be End-to-End(E2E).
Second, I have the opposite experience, where teams seem to want test devs to write slow, brittle E2E tests that require maintaining fragile environments that often need to cross-coordinate among many repos deploying to it (plus any manual changes that always seem to happen).
I see very little emphasis on unit tests. And I see almost none on contract tests, the fast way to break E2E tests into grok-able pieces that can run on every push (just like unit tests).
> let’s shift our focus from output to outcome
I don't really see that a focus on E2E tests outcomes. I just see that as slow, brittle, high-maintenance, and something requiring an environment (thus not running the tests on every push - even slower feedback).
Maybe try to focus on fast, meaningful feedback instead?
- A complicated index of objects in a virtual world. The test creates some dummy objects, moves them around, and runs a consistency check on the data structures after each change. This has caught bugs in the past.
- A lock-by-string primitive, used when I don't want two threads working on the same URL.
- Serializers and deserializers, usually where some data comes in from the outside. Quoted strings in the test cases deserialize, reserialize, and check.
- There are geometry generators which generate spheres, boxes, cylinders, etc. Those all have unit tests.
- Some math functions, such as angle between two quaternions, and some computations around transformation matrices and the viewing frustrum that tell what's visible on screen.
- Some transformations on meshes.
- Some simple parsers.
Mostly things with algorithmic complexity, data format dependency, or non-trivial internal consistency. Hard parts that the Rust compiler won't catch, and which are hard to debug in a larger context.
With those tests in place, and in 100% safe Rust, not much mysterious happens that's internal to the program.
External relationships with the game server are another matter. That's handled with lots of internal checks and logging.
I think this post glosses over the reality that tech workers primarily want to make things work correct, and they generally would write good tests (whichever kind that means for their problem) if they had the time and resources. But the final words seem to indicate doing a check on if the test is valuable... But that's not really where the problem or difficulty lies. It's often an organizational/team barrier (at least in the places I've worked) not a tech problem. And the commentary is often 'well this test is as good as we can do without engaging with all of these known pains'
- the execution time of end to end tests are orders of magnitude larger than unit tests. In a medium ish application your end to end tests will take several seconds to spin up, for example by spinning up fresh databases (even with rollback in gets slow) and other detachable services. A unit test is usually a malloc away and thousands can be run in parallel on your local box in a second. End to end tests easily take hours of compute time time to run in the real world. Good luck running the suite on your laptop. Nothing kills productivity quicker than a test suite that runs slowly
- unit tests are also independent of other code so it’s easy to get some confidence that your code does the right thing by only testing the touched files in your diff before running the entire suite on CI.
- unit tests encourage good code quality since it’s easier to test injectable code. E2E instead incentivises copy pasted test suites that are thousands of lines of code long.
- in the general case end to end tests are asynchronous and require orders of magnitude more effort to run. Asynchronous tests are flaky. Flaky builds slow down your velocity.
- E2E tests rot really fast. One change leads to thousands of broken E2E tests.
- E2E is not a replacement for monitoring your application. The class of bugs you will catch in your E2E suite is a strict subset of the class of bugs you will catch with monitoring and metrics. Build the latter instead and roll out your code incrementally to your users.
- E2E tests are complicated to set up and maintain in ops. Unit tests not so much. Somewhat unexpectedly your development velocity scales pretty much linearly with stack complexity.
In summary (and again context matters and there aren’t any absolutes, but …):
- Don’t do automated E2E tests.
- Do unit tests
- Do metrics and monitoring
You have a limited budget, spend it where you get a good bang for your buck.
Unit tests and smoke/E2E are like peanut butter and jelly, they're better together. And yeah they're expensive to build but it's almost always cheaper than deploying crap and hurting your customers.
Why all the drama?
I think the unit testing "debate" when it comes up tends to be very handwavey, like this article (which I still think is thoughtful.)
Personally I’ve settled on this:
1) Whatever style of tests you write, they need all the love and attention your other code has. You need reuse, abstractions, constant refactoring. This makes me wary of external E2E frameworks in languages separate from your application codebase, but I’m aware most people don’t get to use a single stack across front and back end.
2) Functional tests are great. At the very least your codebase should have somewhere you can see its use cases enumerated (and preferably verified) and it’s rare for this to exist without functional tests (but good on you if you explicitly model use cases in your code). These are the things that actually make you money and should be your primary concern.
3) You can’t test every possible code path in functional tests, or at least not easily and quickly. I’m happy to test the bones of the system end to end but isolate complex logic the same way you would external dependencies. With this isolation you can easily write property based tests to verify the behaviour, and these to me are by far the most (dare I say only) valuable form of unit test.
4) I’m happy for end to end tests to hit mock/fake components but I always mock out the other side of the protocol. That is, tests that query a database always query a real embedded database not a mocked class that returns fake results. Embedded Postgres is extremely fast (20ms overhead in my tests) and perfectly reliable. The same is true of network services running in process on localhost.
5) Any time you mock/fake something you also need some way of verifying you’re not lying to yourself so you should have a minimum number of tests against real dependencies to verify your assumptions.
6) Unit test an implementation too early and you’ve done twice as much work to throw away when you refactor. You need to spike and stabilise based on your judgement. Personally I’m not concerned if I have no unit tests at all in code that is exercised by integration or functional tests, with the caveat about property based tests of complex logic above.
7) If you’re working on a legacy codebase or in a dysfunctional organisation then just do your best, obviously. Most of the time making things easier to test improves their architecture anyway but it’s very hard to pin revenue on that.
If the system is used by humans, then it needs to be tested by humans. Ideally but not necessarily not the customers.
If it is used by just machines then machine testing is fine.
(Per the article, unit tests are clown coding really - nobody cares)
Do we consider webbrowsers to be humans here?
It may be useful for development purposes to create artificial internal 'commercial boundaries', but unit tests are far too fine grained.