Hacker News Reader

> And they can be more brittle than your actual code.

The brittlest tests I've ever seen have been mock-heavy unit tests. Some of them amount to little more than file-change detectors. Those tests can fail massively even when everything still actually works. They cry wolf so often the team ends up trained to unthinkingly change them so they "go green."

> But I think I get what he's getting at in terms of – ideally – wanting to minimize those kinds of tests by transforming previously unreliable parts of the system into something we can use "as a given" (like integer arithmetic).

A noble goal, but practically not a lot different from saying we should aim for formally verified software. After all, it's what we have for computer arithmetic example: http://www.cl.cam.ac.uk/~jrh13/slides/jnao-02jun06/slides.pd... (2006):

> Intel uses formal verification quite extensively, e.g. ... verification of...floating-point unit[s]...

My experience with unit tests is that they force a developer to think through their logic. So the tests themselves,once passing don’t help that much (though a little), but the act of writing them did have high ROI. The coverage metrics also help management understand what is baked.

> The brittlest tests I've ever seen have been mock-heavy unit tests

Not to knock mocking in general, but the need to replace multiple things at once to test one thing is just a big design smell, and I agree: their usefulness tends to be very limited in terms of logic exercised.

Call me a fuddy-dud, but I'm generally working with dependancy injection when doing OOP so stubbing out app layers for test contexts feels kinda natural, and I prefer to make my test mocks the old fashioned way: manually using inheritance. It keeps the pain high if I'm doing something wrong, keeps the tests focused, and means I'm hammering the objects design from the POV of an inheritor/reuser from the get-go.

I've noticed that the additions necessary to make a manual mock tend to be a good predictor of future changes required in the code-base, while framework based mocks tend to spend a lot of time re-declaring things we already know.

>But the more 'real' the test is, the less of your own test it covers relative to all of the other external dependencies.

A strangely underappreciated facet of being a more 'real test' is that you catch more 'real' bugs.

>should your integration test actually make a (test) request to the real live service endpoints?

Ideally you should be able to run your integration test in a mode that mocks it and a mode that calls the real live service so that you can see problems in your code and (and avoid brittleness) and problems with the service, and distinguish them.

> But the more 'real' the test is, the less of your own test it covers relative to all of the other external dependencies.

But the more 'real' the test is, the more of your own assumptions get about how your dependencies work get tested. Assumptions are where most bugs are born and hide.

PS: Sorry for the multiple replies. HN is being glitchy and my previous comment kept getting truncated.

> But the more 'real' the test is, the more of your own assumptions get about how your dependencies work get tested. Assumptions are where most bugs are born and hide.

Very well put—I like this.

> If your code depends on an external service, should your integration test actually make a (test) request to the real live service endpoints?

That's a terrible idea. You've now coupled your tests to someone else's code.

Test the endpoints separately. Wrap the endpoints to allow you convenient dependency injection. Test the wrapper. This wrapper is ideally a very thin layer of boilerplate, that you won't ever touch very much, and shouldn't take much time to write or test.

But this is the key, however: use the data specifying the expected outputs of your third party API as input parameters to the test of the wrapper, and expected outputs from the wrapper as parameters for your integration tests. Keep each set of data in one place, so that changing it for one test, changes it for the others.

By using shared test datasets in this way, you prevent modules from getting out of synch with each other, and can still replace/update parts in separation.

>That's a terrible idea. You've now coupled your tests to someone else's code.

It's a terrible idea not to. If your code isn't coupled to someone else's code your code isn't doing anything useful.

Moreover, you should test against the code your code is coupled to as realistically as possible, which ideally means testing against the real thing.

>It's a terrible idea not to. If your code isn't coupled to someone else's code your code isn't doing anything useful.

Paradoxically, you want your tests separate from the external inputs, because complete control over the inputs is required to systematically and programmatically hit every edge case with every pass of your test suite.

You also need the modularity in your tests and code, so you can fix and replace 3rd party endpoints without being afraid of polluting your code's inputs, or having to rewrite your integration suite. This is where the data contracts come in: they make sure that each module of the system and its partners are speaking the same language, even after they have been replaced or updated to handle new 3rd party behaviours.

I usually have this concept of integration tests covering the interaction of modules you are in control. 3rd party services are tested using something I call "edge tests".

This leads to few mocks, yet I know the expected behavior of the external services.

You're missing a fundamental difference in philosophy. Why should we only care about bugs in our code? They aren't the only bugs our users care about.

A browser bug (for example) may not be our fault, but it is our problem. Therefore, we need to run our tests in real browsers to make sure it actually works, and keep running those tests on new browser releases to find any new bugs. Similarly, mobile apps need to be tested with a variety of different phones.

If you upgrade a dependency and it changes behavior, you're still responsible for fixing or working around it, hopefully before users find out. Lots of interesting bugs happen in the gaps between modules, especially when they're maintained by different teams.

A good reason to test code in isolation is that it makes diagnostics easier. However, if you only do that you're blind to a lot of other bugs.

I beleive our disagreement is fundamentally one of semantics. However, I'll concur that I have not used this method for front end or mobile or embedded code yet, and there may be problems that I haven't anticipated.

That said, I think you miss my point - I'm not saying don't test in a real-world environment: I'm saying decouple your program logic from its inputs, and enforce shared test data between top-level edge and the integration test suites to ensure end-to-end integrity (and do the same for every other part of your code that interacts with another part of your code, fwiw).

> Test the endpoints separately. Wrap the endpoints to allow you convenient dependency injection. Test the wrapper.

So, test in the browser, or on the app. Just test the endpoints separate from your integration test, and use the explicitly stated output of integration test as the input parameters for your browser tests. I'd go far as to argue that browsers and hardware - where you have the greatest concerns about the stability of your 3rd party APIs, are actually the circumstances where separating edge testing from integration makes the most sense.

Perhaps a better way to explain my process is to think of my set of edge and integration tests as the equivalent of your 'integration' test suite, with the addition of specific contracts (the shared data) to ensure that interaction between modules is also controlled for. You are still testing end-to-end and you are still testing with real-world data, since that's what the edge tests will be returning. You are just explicitly stating what you expect at each boundary of the code.

>So, test in the browser, or on the app. Just test the endpoints separate from your integration test

Unless you're testing different behavior when you're testing the endpoints, what you're doing is essentially writing duplicated test code.

That means two sets of tests (add person browser test / add person API test) that will nearly always break on the same bugs. That means two sets of tests to maintain when you change the code.

It's not that you shouldn't sometimes drill down to test code at a lower level, it's that when you do, you should be drilling down to a reusable abstraction that is independent of the higher level code.

> Unless you're testing different behavior when you're testing the endpoints, what you're doing is essentially writing duplicated test code.

Except the endpoints are a wrapper. They aren't doing anything but acting as a pass through for injection. So, you are talking about a few minutes of boilerplate that will only change when the API changes, as a small price to pay for the ability to inject your endpoint dependencies (or their stubs) without the mess and complexity of proxying.

If it's so objectionable, handle injection some other way, without the wrapper: it's not fundamental to the idea of using data contracts to ensure correct communication between modules, while keeping the edge and core integration suites modularized.

>without the mess and complexity of proxying

Proxying is less of a mess and less complex than DI. It requires zero changes to your code and can be done with off the shelf tools, it will continue to work if you rewrite your endpoint in a different language and you easily can do stuff like mimic high load or flaky networks to see what happens to your system.

>If it's so objectionable, handle injection some other way

Why do DI at all if you're just doing it for testing purposes?

What you are arguing for is that code should be loosely coupled. I generally agree with this (although, like DRY, its validity is context sensitive and one can take this approach too far).

This is an orthogonal question to whether code should be tested realistically or not, however.

You want to test your code against real external inputs for additional realism as well as mocks that are as realistic as possible to isolate the external inputs and remove a potential source of test instability.

> ...You want to test your code against real external inputs for additional realism as well as mocks that are as realistic as possible...

Ding ding: it's not either/or, it's both/and across a spectrum from dumb to smart to dumb again.

If you use a google rest API in something serious you want to be able to model the error conditions of that API, you also need to verify the current status and content of that API, you also need to verify your apps behaviour separately for both cases.

Loose coupling and DI are important to strike the right balance, but fundamentally you're either testing your solution or you're not.

>Ding ding: it's not either/or, it's both

Yes, that was the point I was making.

>Loose coupling and DI are important to strike the right balance

DI is useful sometimes (e.g. when you have a set of modules you want to hotswap with one another), but it's often just an overused crutch to deal with the inability of unit tests to couple to real things - like an actual REST API endpoints over a loopback interface.

The whole idea that unit tests drive "good code" by making you DI all the things is a pile of shit. Unit tests just make it painful to not do DI because unit tests are themselves a form of tight coupling.

> You've now coupled your tests to someone else's code.

Your code is coupled to someone else's code. Which yeah, can lead to a lot of problems, but code that doesn't talk to someone else's code is usually not very useful.

> You've now coupled your tests to someone else's code.

And so your tests actually demonstrate working code!

Right, but what I'm saying is, it's even worse than that. Not only are your tests coupled to someone else's code, but your code itself is so coupled.

The trivial realities of the actual world may chill my heart to the bone, but even I can accept this sad inevitability ;)

The thing is, the fact that your code relies on someone else's, and that their code can change out from under you is exactly why you need separation in your tests and modularity in your code. Using the system I am describing, you are still working with 'real world' data - you are just asserting it in advance (since the real world inputs correlated with the inputs of your integration suite via the edge tests) and ring fencing your endpoints.

I've actually been part of a software team driven wild by unknown changes to infrastructure endpoints, causing failure of the integration test suite. And then by the brittleness of an integration suite too tightly coupled to inputs.

I've also worked with test suites that failed when run after business hours, but only during Atlantic DST, for the same reasons. :|

> I've actually been part of a software team driven wild by unknown changes to infrastructure endpoints, causing failure of the integration test suite.

Now imagine that your tests hadn't been so tightly coupled, and there were unknown changes to infrastructure endpoints. Suddenly your code starts failing, and you can't tell why, because your test suite is all green.

"And I haven't found that integration (system) tests have a higher ROI – because they're usually much more expensive to write and maintain."

I think the worst code bases to modify are the ones with heavy use of mocking and dependency injection. You still get no feedback about the behavior when run against the real system and you have to maintain a potentially complex mock.

> complex mock

That's a smell for sure. I've got a codebase with lots of 'mocks' and a good amount of dependency injection, but I don't feel like it's hard to modify. In fact, the mocks and dependency injection make testing possible. But then all of the mocks are really dumb, e.g. implementing an interface in the 'dumbest' way possible.

The most complex mock behavior I have is return a value supplied as a parameter to this object's constructor. The mock then is really just a way to explicitly encode assumptions about (possible) behavior of whatever it is being mocked or injected.

I think the issue you're describing, a massive variance in test and system behaviour, is symptomatic of a poor delineation of the unit/integration test dichotomy. It's almost a skill within a skill, but the moment a mock starts taking on some life or complexity is a moment you should be backing out and refactoring.

Complex mocks shouldn't exist. Mocks of complex interfaces should almost entire be sustainable by your domain objects if they're exposed to your testing 'datalayer'. So if they're required to test, and I've seen many codebases where that's true, then you've generally got a major design issue that is unaddressed in the main codebase.

Personally I use mocks very sparingly coupled with a few strategic stubs (ie hard coded "databases" of domain objects), and almost always code them manually by hand. That way it's very clear if I'm doing something dumb, the sytems DI is leveraged, and the mock is highlighting design issues in a re-use/inheritance context.