Comment by cies | Hacker News Reader

Comment by cies | Hacker News Reader

Interesting article.

I'm a big fan of "economic testing". Test that are not going to "be economically viable" to write during the dev't of new software, usually do not get my permission to be written in that phase.

Unit test thus get written for part that we absolutely want to be maximum certain of that it works correctly. Mostly that functionality can be extracted into a library. For instance a lib that deal with scheduling of events of arbitrary length: we do not want to have issues with those getting messed up, so we try to get 100% unit test coverage of the functionality in that lib.

Another thing with unit test is that I found they mean something different in dynamic languages than in strongly typed languages. In the first unit test are often something to provide confidence that the type system can give you in the latter. Sure both will also test the logic. But this understanding has pushed me to love strong types (as a rubyist), as I can go with less unit test and still be more confident about my code base when it is being heavily worked on. Refactoring becomes so much easier with strong types, as all is "in the code".

When it comes to integration tests I usually want to make a biz case for them. How much does it cost to have it all manually tested, and how much do the automatic integration tests cost to make. While including the benefit of being able to run the automated integration tests at near zero cost (thus several times per day if we need to).

> Unit test thus get written for part that we absolutely want to be maximum certain of that it works correctly.

I think this is a partial answer.

Yes, I write unit tests for such pieces of software. However, what I also write a lot of unit tests for a pieces of software which are hard to write good integration test for because of combinatorial explosion. If I have a part for integration method which consists of a chain of 4 methods, each of which can have 5 different results, that'd mean I'd have to write 4^5 integration tests, which is clearly not possible. If I were to write unit + integration tests, however, I could do with 4x5 unit tests plus a couple of integration test which prove that the overall combination of those methods work well together.

Combinatorial explosions are best mitigated by doing property based testing, design by contract and creating clean, low level abstractions which you test against.

In practice I've also run into a lot of combinatorial explosions for which unit testing is worse than useless. E.g. say you have a library that has two dependencies and interacts with browsers you can have:

* 30 different versions of library A

* 50 different versions of library B

* 25 different versions of the language runtime

* 30 different browsers

That's over a million different potential variations for every test. In practice though, there's a sharp power law. You don't have to test all of those variations, you just have to randomly throw variations at the code until bugs stop cropping up. 99% of the bugs you find probably be found in the first 1,000 iterations.

Fair enough; in this case unit testing is not very useful. I was more thinking about business logic when I wrote that example.

It's also worth looking at all-pairs testing. The insight is that you can cover all pairs of a set of combinations in only O(N^2) tests rather than O(N^D) irrespective of the number of dimensions. The number of tests ends up being the product of the size of the two largest dimensions. So in your example you can cover all combinations of each pair:

  * (lib A, lib B) 
  * (lib A, language runtime)
  * (lib A, browser)
  * (lib B, language runtime)
  * (lib A, browser)
  * (language runtime, browser)

in 50*30 = 1500 tests (or a little more to make the analysis tractable)

This is clever... is the N^2 behavior just due to nCr(n, 2) = n^2/2 - n/2 growing quadratically?

> Unit test thus get written for part that we absolutely want to be maximum certain of that it works correctly.

I hear this kind of rhetoric a lot, but rarely do I hear of code that doesn't need to work correctly.

Code robustness clearly has different priority levels though. A graphical error is much less serious than a part of your code that could nuke your server.

Except that's not a hard rule (not that you implied it was).

A graphical error that prevents users from being able to use the application at all can be basically the same as server downtime in terms of impact to the end user.

If the e.g. scheduling code is messed up, and we find out 5 weeks in, then there is a lot of damage. We need to call all parties to review their schedules with them.

On the other hand if the code responsible for the styling of some menu is not working correct, the functionality may still work, and once we discover the bug we can simply fix it.

> rarely do I hear of code that doesn't need to work correctly.

So there are different grades. That's why I used the word maximum. The cost for these unit test should pay bank, even during the dev't phase, IHMO. There are often areas in a large code base for which this hold true, and often they can be identified as such beforehand; then writing these tests should be a priority.

The difference is not between code that need or doesn't need to work.

The difference is having code for which having some bugs is acceptable, vs having code for which bugs are not acceptable.

There is definitely code for which having some bugs is acceptable: think about some small feature that the user can totally do without. I had to do one recently, and bugs were acceptable, as long as it was shipped in time, and working for the client specific devices. That code shipped with no automated test (testing is done manually), and a few known bugs not worth solving. On the other hand, anything that touch core features must be tested.

> That code shipped with no automated test (testing is done manually)

An issue with that is bus factor when it needs to be changed in the future. What if the original dev isn't around any more?

There is a test plan, and specs. Acceptance tests (manual) weren't even done by a dev.

Also, the code has been reviewed and is documented.

Bus factor really isn't a problem in the instance I was talking about. In fact, the original dev won't be around in a few months, and that's fine. The rest of the team will be able to modify the code.

Reply on news.ycombinator.com