Comment by molf | Hacker News Reader

Comment by molf | Hacker News Reader

Parent Full thread

I’m not sure this is good advice. I prefer to test as much of the stack as possible. The most common mistake I see these days is people testing too much in isolation, which leads to a false sense of safety.

If you care about being alerted when your dependencies break, writing only the kind of tests described in the article is risky. You’ve removed those dependencies from your test suite. If a minor library update changes `.json()` to `.parse(format="json")`, and you assumed they followed semver but they didn’t: you’ll find out after deployment.

Ah, but you use static typing? Great! That’ll catch some API changes. But if you discover an API changed without warning (because you thought nobody would ever do that) you’re on your own again. I suggest using a nice HTTP recording/replay library for your tests so you can adapt easily (without making live HTTP calls in your tests, which would be way too flaky, even if feasible).

I stopped worrying long ago about what is or isn’t “real” unit testing. I test as much of the software stack as I can. If a test covers too many abstraction layers at once, I split it into lower- and higher-level cases. These days, I prefer fewer “poorly” factored tests that cover many real layers of the code over countless razor-thin unit tests that only check whether a loop was implemented correctly. While risking that the whole system doesn’t work together. Because by the time you get to write your system/integration/whatever tests, you’re already exhausted from writing and refactoring all those near-pointless micro-tests.

> I’m not sure this is good advice. I prefer to test as much of the stack as possible. The most common mistake I see these days is people testing too much in isolation, which leads to a false sense of safety.

You make it sounds as if the article would argue for test isolation which it emphatically doesn't. It in fact even links out to the Mock Hell talk.

Every mock makes the test suite less meaningful and the question the article is trying to answer is how to minimize the damage the mocks do to your software if you actually need them.

But ultimately it suggests this test; which only tests an empty loop?

  def test_empty_drc():
      drc = Mock(
          spec_set=DockerRegistryClient,
          get_repos=lambda: []
      )

      assert {} == get_repos_w_tags_drc(drc)

Maybe it's just a poor example to make the point. I personally think it's the wrong point to make. I would argue: don't mock anything _at all_ – unless you absolutely have to. And if you have to mock, by all means mock code you don't own, as far _down_ the stack as possible. And only mock your own code if it significantly reduces the amount of test code you have to write and maintain.

I would not write the test from the article in the way presented. I would capture the actual HTTP responses and replay those in my tests. It is a completely different approach.

Yes, because it's showing how the simplest-possible mock already gets ugly if you don't follow the advice. Making the example more complicated would dilute the point it's making.

The question of "when to mock" is very interesting and dear to my heart, but it's not the question this article is trying to answer.

My take on unit tests is that they happen "automatically" for the things that actually need unit testing. Consider a load shedder, a parser, or an implementation of an algorithm. Most devs will naturally write a unit test for anything with complicated logic just because it's easier to test complicated things in isolation.

Reply on news.ycombinator.com