Hacker News Reader

Agree. Consider the situation of data entry professionals, those people who transcribe audio recordings or enter data into databases.

If you have one person entering data into a computer, then the odds of them introducing an error and failing to spot it are fairly high.

If you have twice as many people entering twice as much data data, then the odds of an error getting introduced are roughly doubled.

However, if you have those two people entering the same data, then their mistakes cancel each other out. If person A and person B both entered the same thing, it's extremely unlikely that it's incorrect. If they differ though, the a problem has been identified, and can now be fixed.

The odds of both of those people entering the same piece of data incorrectly is tiny. Likewise, accidentally introducing a bug into both the production code and the test is pretty unlikely.

That said though, if those two theoretical data entry people above are given the wrong data to enter, then they the system cannot protect them. They will both correctly enter incorrect data. "Garbage in, garbage out".

Likewise, if the requirements of a piece of software are poorly understood, then it is quite likely that both the test and the production code will implement the same "bug". Writing tests won't fix a failure to understand the problem you're trying to solve. And they're not supposed to.

> The odds of both of those people entering the same piece of data incorrectly is tiny.

This is just not correct. There may be a systematic reason why they are making a mistake (e.g. a miss-pronounced word) in which case increasing the confidence intervals does not increase the accuracy. Check out the concepts of accuracy, precision etc from physical sciences.

Tests don't exist for solving bugs as much as they exist to prevent regression. Writing exhaustive tests is not sufficient to bring c' to zero, because some of those bugs exist because of the developer's own presumptions about how code should behave.

But testing exhaustively means the full range of inputs are used, and therefore, if a bug arises as inputs not considered by the dev initially, it would show up as a failure.

What you're actually saying is that reaching full exhaustive testing is near impossible.

I think what they're saying is that both the code and test will contain the bug, since the bug is in the programmer's understanding rather than just in their coding.

If I understood func(5, 6) should return 9, but the client actually wants 10, no amount of tests I write will reveal the error.

I don't think that any form of formal specification will solve that issue. The only way to minimize the impact of it is to work on small iterations (agile) and be close to the client. One of the weakness in agile methodologies today is that the client representation (Product owner) is not aware of the client needs or too far from the client point of view.

This is roughly what I meant - there are ways to end up with zero bugs, but writing tests is usually insufficient for achieving such a goal.

Testing for regressions is a a valuable thing to do, but your point is also valid. We can prove that it is impossible to test every scenario in general (for example, does this program end ;-) ). In my mind, unit tests exist for documenting what the code is doing now -- not to determine if it is correct (acceptance tests are different). Your goal for a unit test is to have it fail when the behaviour of the production code fails. This way you can verify that the changes you are making are having the effect you expected. Similarly, you can be warned when changes you make affect part of the code you didn't expect to be affected.

Of course, you will never get full coverage, but tests are a programmer's tool. It allows you to reduce the amount of time you spend inspecting parts of the code you aren't working on. For example, when I'm writing code with good test coverage, I can find out what happens if I change an attribute in a structure simply by doing it and watching the tests fail. Each failure tells me where in the code I'm likely to have to make a change.

If you make a small viewpoint change that tests are not to find bugs, but rather are a tool to help you understand the code base, they make a lot more sense. (IMHO)

Bugs in tests exist (and are just as prevalent as bugs in production code in my experience). However, writing production code introduces a kind of constraint on the system. If you run a function with certain program state, then it returns a certain result. No matter what you do, this will happen because that's how you wrote the code.

A test introduces the same constraint. Instead of implementing the function, it runs the function. No matter what you do, the test will run certain code and get a certain result. Tests pass when the constraints introduced by the production code match the constraints introduced by the tests.

Because of this, if there is a software error in the production code, usually you need to have a corresponding error in the tests in order for the test to pass. This definitely happens from time to time, but the odds of it happening are much lower. Similarly, if you have a software error in the tests, you need to have a corresponding software error in the production code in order for the test to pass -- assuming that the test is actually exercising the code (something you can't always assume, unfortunately).

The other fault in that reasoning is that usually test code is much much simpler than product code. It's a lot harder to write bugs when the structure of the code is just 1. call single function with some input data 2. verify that output data/side effects meet expectations 3. repeat for more sets of inputs.

One of the mistake I see a lot around this is when people start to write intelligent mocks. Usually this is where test bugs reside.

The best is when someone writes an entirely new implementation of something just for tests, and only calls that from their tests instead of the actual product code!

Then you just get blank stares when you try to explain why what they've just done is completely pointless.

The way they do it may be pointless, but the intent is not. The crux is making sure your two implementation are independent. There are two ways to achieve this. The first is clean room. Just have independent persons or teams implement the same thing. With different tools and programming languages if you can. The second is use an entirely different approach for both. The test code for instance can be constructed around the tests, and for instance rely on the fact that everything happens at the same place at the same time (when you run the test).

---

I'm currently writing on a secure handshakes generator. https://github.com/LoupVaillant/Monokex It generates specifications and source code from a Noise pattern. A bit like Noise Explorer, only with less features. Previous versions of the code were hand written, and the generated code is doing its best to look like it was hand written. Here's the result: https://github.com/LoupVaillant/Monocypher-Handshake/blob/ma...

Now how do I ensure the code match the specs? (The correctness of the specs itself is currently checked by hand.) I could write another implementation, but I'm only me, and I can't go erase my own memory to make a clean room implementation and compare the two.

Instead, I took the specs, and wrote code that takes all the inputs, and spits out all the intermediate buffers and outputs, paying no heed to stuff like order of execution, or who does what. I only concentrated on generating the test vectors: https://github.com/LoupVaillant/Monocypher-Handshake/blob/ma...

The structure of the vector generating code and the actual production code are very different. This is how I decorrelate mistakes, and make sure that if the two "implementations" agree, I'm very likely to have something that works.

---

Of course, this is all a lot of effort, so I wouldn't do this for non-critical code.

I think this holds for well written tests, but if one is not talented in writing them then more tests will simply bolster the bugs in the main code. For example, if a developer generate test data by running the program and copying it's output (something I see far more often than I'm comfortable with), they can tell themselves that section was tested thoroughly and must be bug free. Thus when someones notices an end result someplace is wrong, the first place checked may not be that "well tested" function. Tests are only as good as the domain knowledge of the person writing them.

Do you have an actual source that corroborates that with actual data? It seems true, but I haven't found anything that actually tests the premise. I've found studies that show that code with lots of unit tests tends to be more bug free, but not anything about whether unit tests increases (or decreases) development time or whether test code actually has less bugs. I think it would be interesting if there is also a measure of code complexity, and if less complex test code correlates with less bugs in the test code.

Arguing from the point of logic and formalisms the author is talking about, this would seem to be true.

> Instead, it reduces that number to c' * N with c'<c and the difference being caused by your test coverage.

That is unproven conjecture. It feels right, but just because it is written as a formula doesn't automatically mean it is correct.

It is an axiom.

If your testing is such that this isn't true then you are doing testing wrong, in a self-evident sort of way.

That's about as useful as observing that if your code is such that it doesn't produce the correct result then you've written the code wrong, in a self-evident sort of way.

> observing that if your code is such that it doesn't produce the correct result

Hmm. Sounds like you are doing some sort of testing.

To expand on the other commenter:

If adding test code makes your code under test more bug-prone, something has gone horribly, terribly wrong.

Maybe we have a disconnect in our definitions of "testing".

How do you define the meaning of it, and could you illustrate how it could, on average over a code base, increase the number of bugs in the product code under test?

Assuming the same time constraints, time spent writing tests competes with time spent writing product code. Ideally, tests save more time than would otherwise have been spent debugging and/or save time and money by catching bugs early.

In bad cases you get no return on the time spent writing tests and simply end up with less product code (similar bugs/LOC, so fewer bugs and reduced features). In terrible cases you end up with rushed product code and more bugs/LOC.

Yeah, it's like saying "CAP says partitions decreases availability/consistency, so putting your app servers in a scale group will make things less available/consistent."

That's really superficial analysis. Yes there is some fundamental tradeoffs. But it is possible to change the properties of a system or combine them in intelligent ways and move the curve.

Or it's like saying that speedometers make your GPS less accurate due the Heisenberg principal.