We need to talk about testing

The saddest thing about articles like these is that they are so full of wisdom, but they can never seem to compete with the "hot new methodology."

Like it or not, the "agile" movement had a huge impact on the way people do their jobs, in part because it had a name, it had advocates, and it had principles that you could fit on a PowerPoint slide.

But good testing (like good design) just can't be encapsulated in a handful of principles you can teach to a newbie. Good testing requires wisdom.

I can write a lot of words about what I've learned about testing over the years, but it probably won't do a good job of conveying that wisdom to others. The best testers I've ever worked with have a strong gut instinct, developed by experience, for where the bugs will be found, and how to effectively spend our limited time-budget for tests.

Is it worth adding more end-to-end UI tests? What if they run slowly? (How slowly?) Is it better to add more unit tests? What if adding unit tests requires aggressively mocking out dependencies? Which parts of the product require more testing, and which ones have been adequately tested?

These questions don't have quick, easy answers, but the quick, easy answers keep winning mindshare, and we're all impoverished as a result.

Which leads into being very clear about what you want to achieve with testing, correctness, robustness, fit for purpose etc, and be clear how much effort you want to put into each area. Often I have seen people put a lot of effort into testing things that really doesn't have much payback compared to what they could be testing instead. So be careful of the opportunity costs of your testing efforts.

See also "We're all [person] right now."

If you have a system to deliver software updates in one hour, you will use it to ship 'just in time' updates. Structurally it is hard to argue against fixing the customer pain when you can easily deploy a fix.

But the long term effect of having CD is that testers get pushed out of the system.

If we deliver code everyday we can no longer afford to spend 3 days testing each release. If we can't spend 3 days testing for defects in the entire product, we end up only testing a small amount of the actual product on each release.

If we deliver code multiple times per day we probably cannot afford to do 'any' manual testing.

CI/CD encourages spending as little time as possible on testing. If deploying code takes 10 minutes, why let testing make it a 3 day process?

The long run effect is that the quality of software decreases in general. Bugs are continuously deployed into the system and you never reach a state of doneness.

I'm personally a huge proponent of End 2 End Testing. You can think you covered everything in isolation, but the user is presented with bugs nonetheless. I have near 100% E2E test coverage. Yes it takes time to run, yes third-party integrations can result in failed tests. (Which probably signal an issue with you or your third-party anyway). I believe a full E2E test requires less maintenance and has a broader coverage than Unit Tests.

For instance I run TestCafe E2E tests that tests all possible user interaction from signup and paying for a subscription to account termination. I go as far as reading expected emails through a web email client and verify the content in the received email using TestCafe. I test cache invalidation, a host of automated member expiration, warning and authentication / security scenarios. All this should be done anyway, why not do it using E2E tests. I did a rewrite, switching from Vue.JS/SSR to React on Next.js I had only minimal code changes to my E2E tests. And with minimal I mean a couple of ID/Class DOM references to account for different components in Vue vs React.

Not only do you test code paths, but also all your components in the stack and the interaction between them, which for me are Node.js, Nginx, PostgreSQL, Redis, Amazon SES, etc. If something fails, you will have to do some digging, more so than a failed Unit Test. But more often than not you know what has recently changed and so where to look for possible issues. Your E2E tool makes a screenshot of the browser fail-state often highlighting the issue at hand.

Of course these types of tests work best on websites, and not so great on say low level GPU driver code. They may also become to cumbersome and slow on highly complex sites. The biggest drawback is probably the time it takes for a full test run which in my case can take up to 20 minutes in a headless browser.

I think keeping this in mind is important as a product and codebase scales since a lot of testing approaches shift in importance over time. The articles sidebar about TDD is a micro-example of this, but there are ones on the order of months/quarters/years that Ive come across as well.

One example is how you likely only want end to end and integration tests early on in a development lifecycle, rather than excessive subsystem tests and mocks. This is because your end to end product experience is likely to change less than the subsystems that implement specific features (eg user auth). Over time, as your subsystems solidify you’ll likely want to reprioritize and keep end to end tests lower in proportion since they take longer to execute. In other words, your test suite should be proportional to the expected requirement changes over time.

Another more out-there example is the disabling of tests over time as test coverage starts to overlap. A lot of tests in a large codebase exercise the same codepaths, and this is usually not a net benefit (it sometimes can be, eg if one test runs really fast vs another more comprehensive one thats slow). So having a way to evaluate whether a test is actionable or useful is something that emerges organically as large codebases begin to change.

On the other hand, I'm a little confused at the idea of non-automated tests that provide value. If you're interested in security/compliance/accessibility/etc., those seem like general software quality things, but they don't seem like "tests." To me, a "test" is a thing that either passes or fails, by analogy to tests in education, tests in law, etc. It's definitely valuable to review code for security issues or ensure that you're taking accessibility into account, but unless you're asking a question that can produce a yes or no, then you're not increasing stakeholder confidence through evidence, per the article's formulation (which I agree with). If you think there's value in manual testing because a human will provide you with "insights and feedback," great, but if you do hallway "testing" and everyone successfully completes the task but provides no feedback, is that a failure? If you can't find any security bugs from reading the code, does that mean there are none?

If you want unstructured information from humans as part of inputs to your design, great (and that's a great reason to shift feedback left), but I think that's a different category from testing.

You can't automate everything but you can automate a whole lot. Static analysis tools, better languages, modeling tools, etc. can find or prevent security problems more reliably than a human. Linters can check for accessibility issues (missing alt tags, missing ARIA attributes, etc.), and you can write integration tests that try to drive your entire product using only accessibility APIs to make sure your workflows are covered. Automated tests can look at emitted logs and metrics and make sure they're emitting what you expect and not emitting what you don't. And so forth.

This is a good, succinct description of why you should write tests.

Though, as a solo developer who's considered sinking some time into writing tests for Report Card Writer, I've chosen not to. No amount of testing will help me make sales, which is my primary problem at the moment.

One thing that I believe that’s also left out here is that Test-driven developed code looks different than code that isn’t. All the TDD code is per definition testable but other code isn’t always testable. (Mind you, I say code here and not programs.)

I also feel like there’s a missing point in the article, which is how easy something is to test. If it’s not easy to test something or setup a test for it it will probably never happen. I think you should write systems to be easily testable since then people will do actual testing. And if you don’t test that continuously (that it is testable) you will end up in a system that is eventually harder to test. Automated tests will not happen for something that isn’t easy to test and automated tests are kind of a way to acknowledge that “well, at least this works”.

A lot of my stuff has both. If it is heavily UI-centric, or involves device interfaces and/or communications, I tend to use test harnesses.

Unit testing is good. Combined with code coverage tools, it can be quite useful.

But having unit tests is not any kind of quality assurance. It means that the "low-hanging fruit" has been picked.

Test harnesses and "monkey testing" require a lot of discipline, but are often well worth it.

I write about the approach I take, here: https://littlegreenviper.com/miscellany/testing-harness-vs-u...

A different take is that we want tests to have two qualities - correctness (the test cases should match the desired program behavior) and comprehensiveness (the test cases should cover as much of behavior as possible). The only way these are both 100% attained is if the source of your test cases is effectively another implementation of the same specifications, and you compare the behavior of the two.

The comparison itself can be automated, but the creation of the other implementation cannot. So, logically, if you want to automate it anyway, you have to compromise one of the two qualities, either being less correct (be less strict when comparing program output) or less comprehensive (verify less possible inputs).

So it seems to me that the "automate all the tests" folks want to have a cake and eat it too. They want to have an automated comprehensive and correct test suite, without the effort of writing an approximation of another implementation (used to compare).

In the past (I love how the blog post says "I am not advocating to returning to the Dark Ages of software delivery", as if that was necessarily terrible), you had a QA team to do exactly that - approximate another implementation of the same program directly from the specs. The better the approximation, the better the verification. But the cost is duplication of effort, in some sense. If you try to remove the duplication (i.e. for example by having the same person create both at the same time), you're likely to compromise one of the two qualities, without realizing it.

Let me state the above yet differently. The "let's automate testing" approach is based on the assumption that human tester is running the same tests over and over. But that's not the case, the manual testing is actually different each time, so what you invisibly lose by automation is comprehensiveness.

In fact, the QA's job in the past was to have another person (other than the developer) trying to make the sense of the specification (and presumably approximate the implementation with the test design), and check if the specification was translated correctly by running a comparison of their understanding to the developers'. While the comparison itself can be automated, the another look is important to discover the parts of the specification are actually weak, and can be understood differently. I don't think testing a program is solely about its intrinsic properties, but rather about checking the correctness of the translation from specification to the executable code.

# Test your tests

## Summary: test interfaces, not implementations

Tests that pass when something is actually broken, and fail when things actually do work, are worse than useless- they are positively harmful.

## The requirement

Let's say we've been tasked with returning `400` when `GET` `/users/<userId>` is called with a negative `userId`.

## The test

The requirement can be turned into a test that hits the endpoint with a negative `userId` and checks that a `400` is returned:

```java @Test public void getUser_InvalidUserId_400() { expect().statusCode(400).when().get("/users/-1"); } ```

## Implementation

A ubiquitous style of implementation may look something like this: (ignore whether you like the style of implementation or not, it's just an example)

```java public class UserResource {

    @Inject
    private UserService userService;
    
    @GET
    @Path("/users/{userId}")
    public Response getUser(@PathParam("userId") final Long userId) {
        try {
            return Response.ok(userService.findById(userId)).build();
        } catch (final UserException e) {
            return Response.status(e.getErrorCode()).build();
        }
    }

```java public UserService {

    @Inject
    private UserDao userDao;

    public User findById(final Long userId) throws UserException {
        if ((userId == null) || (userId <= 0)) {
            throw new UserException("Invalid arguments", 400);
        }
        return userDao.findById(userId);
    }

```

## Another test

Everyone knows a test hitting the endpoint is not enough- more tests are required. A ubiquitous style of additional test may look something like this:

```java public class UserResourceTest {

    @InjectMocks
    private UserResource userResource = new UserResource();

    @Mock
    private UserService userService;

    @Test
    public void getUser_InvalidParams_400() throws UserException {
        doThrow(new UserException(INVALID_ARGUMENTS, 400)).when(userService).findById(-1L);
        assertThat(userResource.getUser(-1L).getStatus(), is(equalTo(400)));
    }

## Test the tests

### Break something

Let's say the negative `userId` check is removed from the `UserService`. (by mistake or whatever)

The test that hits the endpoint will fail, because it will no longer get a `400` when `GET` `/users/<userId>` is called with a negative `userId`. This is exactly what we want out of a test!

The additional test however will pretend the negative `userId` check is still in the service and pass. *This is literally the exact opposite of what we want out of a test!*

### Refactor something

Or, instead, let's say the `UserResource` is refactored to use a `NewBetterUserService`. (the `NewBetterUserService` still throws an exception on negative `userId`)

The test that hits the endpoint will pass, because it will still get a `400` when `GET` `/users/<userId>` is called with a negative `userId`. This is exactly what we want out of a test!

The additional test however will fail because it expects the `UserResource` to call the (old) `UserService`. *This is literally the exact opposite of what we want out of a test!*

### But... The `UserServiceTest` would fail

Maybe there's a `UserServiceTest` that would fail if the negative `userId` check is removed from the `UserService`:

```java public class UserServiceTest {

    @Test
    public void findById_InvalidParams_400() throws UserException {
        expectedException.expect(UserException.class);
        expectedException.expectMessage("Invalid arguments");
        assertThat(expectedException.errorCode(), is(equalTo(400)));
        
        userService.findById(-1L);
    }

It's irrelevant. The `UserResourceTest` test is still a bad test, because it fails on refactoring. And even if there is a `UserServiceTest` that fails if the negative `userId` check is removed from the `UserService`, the `UserResourceTest` doesn't fail if there is no such `UserServiceTest`, or if there is but it's incorrectly implemented etc. It just pretends the negative `userId` check is in the `UserService` and passes.

If you were absolutely adamant about testing the implementation (which you shouldn't be, because implementation tests fail on refactoring, making them bad tests), the "correct" way of testing that would be:

```java public class UserResourceTest {

    private UserResource userResource = new UserResource(new UserService()); //not a mock

    @Test
    public void getUser_InvalidParams_400() throws UserException {
        assertThat(userResource.getUser(-1L).getStatus(), is(equalTo(400)));
    }

Because now, if the negative `userId` check is removed from the `UserService` the test will fail. But since it also fails on refactoring, it's still a bad test.

### But... Mocks are useful

Yes, mocks are useful, and they do have a place. Eg, instead of connecting to a real db, the DAO methods could be mocked, and REST calls to other services etc could be mocked too. They're external dependencies not part of the core of the system under test. The case made here isn't that mocks are never useful or always bad, it's that mock testing is often used excessively for internals of the system under test, which doesn't make sense.

Also, I'd still argue that, by default, there should still be a strong preference for, rather than mocking DAO calls to db and REST calls to other services, actually spinning up a real in memory db (eg [https://github.com/vorburger/MariaDB4j](https://github.com/v... lets you do this), and actually WireMocking calls to other services. Then, more of the system under test is being tested, with less effort.

Ie unlike with regular mock testing, it's verified that the app doesn't just call a DAO method but also that DAO method actually connects to the db, runs the expected query and returns the expected result based on the primed content in the db, and that the app actually hits the configured external service url with the expected request and handles the primed response based on the Wiremock stub etc etc.

While such tests take a bit longer to run, you write far fewer, higher quality, less brittle tests, because you're not having to write and maintain `nn` mock tests between every layer of your app internals to check every single call on the way through the stack. You can freely refactor literally all* the internals all you want without having to change the tests at all - they will keep passing as long as everything works and they will fail if something doesn't. (which, again, is exactly what you want)

## Terminology

The test that hits the endpoint is commonly referred to as a "integration test", and the other test is commonly referred to as a "unit test".

Kent Beck (the originator of test-driven development) defines a unit test as "a test that runs in isolation from other tests". This is very different from the definition of a unit test as "a test that tests a class/method in isolation from other classes/methods". *The test that hits the endpoint satisfies the original definition.*

But it doesn't really matter if you want to call a given test an "integration" test or a "unit" test. The point is the test fails when something breaks and passes when something is improved. If it does the opposite, it's not a good test.

## More on the topic

* http://googletesting.blogspot.com.au/2013/08/testing-on-toil...

* http://codebetter.com/iancooper/2011/10/06/avoid-testing-imp...

## Misc

There are other problems with the example, such as eg

* Primitive Obsession: using a Long to represent `userId` and using procedural inline checks in the service to check for negative `userId`. Instead, create a class `UserId` that encapsulates and enforces its own checks.

* Hidden Dependencies: `UserResource` has a hidden dependency on `UserService` and `UserService` has a hidden dependency on `UserDao`. Dependency Injection frameworks like Spring encourage such hidden dependencies, and the excessive use of mocks in unit tests.

* The service shouldn't know about HTTP error codes.

These are simplifications in order to keep the example short - the scope of this article is testing.