Selective Unit Testing – Costs and Benefits (2009)
blog.stevensanderson.com
blog.stevensanderson.com
I tend to find that higher level integration or end to end tests are a better approach at dealing with this type of code.
Whilst in theory it is better to separate the coordinator from the algorithm, in practice when somebody throws a huge muddy code base at you it's better not to start tearing up the insides until you've got something that gives you confidence that you haven't broken anything when you refactor.
Unit tests present something of a chicken/egg problem in this regard. In such cases you need to refactor heavily to be able to use them. You will need refactoring safety to do this, however. They don't provide refactoring safety until you do it, though. Dilemma.
On the two projects where I followed the "high level tests first" approach I ultimately ended up not decomposing after that anyway since the integration tests ended up being sufficient and all but eliminated the pain of having un-decomposed code.
That is, I could have safely rewritten it so that it was "unit test friendly" but I started to wonder if tearing up the insides of a code base in order to accommodate the intransigence of unit tests was really such a great idea after all given the risk and extremely high cost and the fact that integration tests did the job they were supposed to just fine.
It's a controversial topic though. Not everybody agreed with me on this. Some people view unit tests as inherently "cleaner" (not sure why) and others are militant about a test taking 50ms being so much better than a test that takes 5 seconds that tearing up the insides was justified. In both cases, I suspect it was a case of principles overriding the underlying cost/benefit analysis.
Likewise on the controversy - my favor toward integration tests tends to get replies in favor of unit tests instead, in which case I'll say I understand their benefit and use them too. E.g. sometimes a method or class that handles some computation needs more coverage than what an integration test typically provides. Maybe the class or method gets called within an API, where inputs or contextual data are well understood. But it might also be called from back end jobs, or by code written by internal data teams where inputs or related data are more nuanced. In such cases, it's worked well to mock the class and go to town with various input or edge case data that might fall outside the perview of an end-to-end test.
Seems like an innocuous and trivially true statement, but on some teams which follow SOLID like a religion it's heresy
Something that cannot be tested is something that you cannot know if it works or not.
Perhaps “better” has a different meaning for you?
A good rule of thumb is that if you need heavy mocking to get unit testing done, you should probably just integration and end-to-end test it.
Example 1: typical machine learning training microservice. You query a pile of data from a few DBs, join it and process it, train a model on it and evaluate it.
You could mock the DB queries to test it, but such a mocked unit test alone is dangerous (tests could pass when the relevant DB tables aren't updated correctly IRL) and doesn't add much of anything. Just integration and end-to-end test it.
A machine learning inference service OTOH is much more natural to unit test.
Example 2: you have an simple endpoint microservice with an internal caching structure (just for efficiency purposes).
The "unit" here is still the user facing endpoint, trying to write unit tests for the internal caching structure will tightly couple your internal API between the caching layer and the API facing service. This makes refactoring harder for the service will adding little upside for what's useful overall here.
You should unit test the cache THROUGH the unit tests of the endpoint service in this case if possible.
SOLID people tend to cry at "unit testing through something else", but this is actually what Kent Beck really meant by "unit" in the original design (a unit is at the software system level, not in the implementation level of a class being a unit).
[Controller] -> [Service] -> [Repository] -> Actual DB
I've found integration tests highly valuable at the Repository level. They've helped me not only catch bugs, but also guided me though migrating to a completely different DB. So, I am really happy with that choice. Unit test + mocks would have been of nearly zero value here.
I've found DB-isolated tests beneficial at the Controller and Service layers. But keeping the tests for these two layers separate was kinda silly. Ultimately I want to know that the controller + service yields the result the consumer wants. For the sake of validating the logic in these classes, why would it matter if it took path A or B? If I had put these tests together, the ration of tests to lines covered would've been way less, with a relatively minor impact on readability.
And sometimes that's okay. Suppose you are partnering with a SAAS to release their alpha API as their first client. Maybe don't bother writing too many mocks against the SAAS, because the API is likely to change really quickly. You cannot know if it works, and that's okay, because there is generally not an expectation that it should continue to work, even if you knew it worked today.
So let’s instead talk about other dimensions:
Are you verifying that your application conforms to its requirements?
This is something that is usually done on a System Under Test the size of at least an application module, a service or set of services. In some cases, for legacy systems, I have even targeted the UI with these tests.
Normally for this type of testing the SUT cuts out third-party dependencies to make things controllable - instead you point it at semantically equivalent but shallow fakes for these dependencies.
These are nice tests that are quite stable over time and resilient to changes in implementation details, allowing you low-cost refactoring.
These tests are good drivers for technical tests - the tests that support the development process itself.
These typically target smaller Systems Under Test. For example, checking a calculation function, checking that you can save and load all documents using a generator and property-bars testing etc.
If you let the high-level requirements tests lead my experience is that fewer of these low level technical tests are necessary than a bottom up TDD approach would lead you to.
The low-level “unit” testing with micro units (classes, methods, functions) is the “wax on, wax off” of test driven development. It exists to establish a habit and then as your experience grows you can combine methods and eventually select the solution that fits best on the exact context.
I think Alistair Cockburn called this shu-ha-ri levels of development.
Tests are not masturbatory method of improving your self-esteem as developer, but a method to write robust code, easy to refactor or modify without breaking unexpected things. Any approach to testing should be motivated by business use case - does this approach actually lower expected future development cost in terms of fixing bugs and adding more functionality? How much it will cost if this part of system breaks?