Hacker News Reader

Comment by arkh | Hacker News Reader

You don't want to test code. Coverage is mostly useless.

You want to test functionalities: setup end-to-end tests. It will be hard, it will take time and it often is a solo endeavor for the first months. But when tested functionalities stop breaking for no reasons even when you refactor a lot of code, when "the project for which a dev put tests" starts being "the project we are confident to push to prod without breaking things" you should get some traction.

And once you think all your functionalities are tested you can measure code coverage: what is not covered should be dead code you can remove.

View on HN

Pointing people only at end-to-end testing seems poor advice. There is value in a mix of all sorts of testing. I agree that testing functionality blows counting coverage by the line out of the water. But since a lot of code that implements algorithms, string manipulation, etc. can be tested just fine with unit tests, and since end-to-end tests are expensive, easily more brittle, require a lot more dependencies, it would be better to have a healthy mix of different types of tests.

The comments was specifically about the use case of getting a large refactor merged.

If you are simply adding code or making simple changes, then unit tests can have advantages. It is when you are making a large refactor that you need end-to-end testing to verify that nothing was broken.

I can't get behind this.

End to end tests are indeed important.

But let's say you have an end to end test that covers A, B, C, D, E, and F.

One day this test stops working. Well, what's the culprit? A? B? C? D? E? F? If you had more granular tests in addition to the end-to-end tests, you'd know.

Now, I agree there's a possible downside here. Maybe there comes a time when C and D are no longer being used. It might not be obvious and C and D could linger in the code base, with corresponding dead weight in the test suite. This is not ideal. However, I would rather have this than a tough-to-debug test failure.

I see this over and over also in organizations I work in and even have to fight it in my own behavior..

If you run an end to end test automation and it fails and then you are afraid or otherwise hesitant of doing the process manually you are out of touch with the system and probably missing the forest for the trees when you work on subsystems.

While I would like to have better end-to-end tests, I know that I will have to fight more problems of getting too far away from how the system works.

> Well, what's the culprit? A? B? C? D? E? F?

I honestly don't care. I know this merge is not touching production. I could have all of A, B, C, D, E and F test suites working individually if they break when put together they are all bad.

The developer have tests so they know exactly how to reproduce the bug. Time for them to earn their salary.

> Well, what's the culprit? A? B? C? D? E? F?

Probably, the one you just changed. How often is it really going to be that hard to debug?

I only changed L, which is nominally unrelated to all of those. You thought this was a clean, well-modularized codebase?

    You thought this was a clean, well-modularized codebase?

Well, even if it's nice and clean A, B, C, D, E are each going to have their own dependencies!

If you change B and the A/B/C/D/E integration test breaks.... duh, it's obvious. Yes, that's easy.

But A, B, C, D, E are each going to have their own dependencies.

You could change X, Y, and Z and now the A/B/C/D/E integration test breaks. Which one was it?

Or maybe you're super cautious and you only change Z. And now the A/B/C/D/E integration test breaks. Obviously it's Z. But how? Why? It's a four-developer team and you're only really familiar with R, S, T, U, V, W, X, Y, and Z.

There's also the fact that unit tests serve as living, de facto documentation on how to actually use the classes. You can both use and iterate upon Z a lot faster and more confidently if it has its own tests.

The overuse of overly granular services can make this pretty awful unless you've put event tracing in place.

I mean, the obvious answer is don't do that, but this is the world people are embracing.

Coverage is very useful in determining whether your tests actually do what you think they're doing. It's not very useful as a metric to be optimized. Writing good end-to-end tests that really cover the functionality as specified is very difficult, doing so blindly without tool support is doomed to fail.

An even better tool than measuring coverage is setting up a mutation testing framework and adding tests until all mutations are caught.

I agree, and I think there is an arrogance by those with the view that 'coverage testing is useless' which blinds them to the real purpose of code coverage testing: to tell you what you haven't tested yet.

Coverage isn't supposed to tell you anything about the quality of the tests beyond what code simply has never been tested yet.

And yet, people seem to think "code coverage testing" means something else entirely. The only purpose for paying attention to coverage metrics, as a manager, is to understand how much more work there is to be done on the tests - or, to determine code paths which never get executed and are therefore dead weight on the project.

> the real purpose of code coverage testing: to tell you what you haven't tested yet.

I can get 100% code coverage with tests passing on:

    // Returns the result of A*B
    int mult(int A, int B) {
      return 4;
    }

I won't know the functionality is not tested. Code coverage will just be an illusion of security.

Code coverage is not all you should be doing.

Tests need to cover small units one at a time. I've been bitten too many times by tests that were too complex and attempted to test too many things at once. They invariably ended up doing nothing of value. Test setup becomes overly complex and prone to oversights. Writing another test for a variation of a problem that takes a slightly different code path becomes a chore. It also gets hard to spot that odd untested corner case hidden deep inside some complex logic.

The proper way to think about this is to have unit tests that cover the nooks and crannies of the business logic and integration tests that make sure that interfaces are correct.

Reply on news.ycombinator.com