Hacker News Reader

Comment by hamdouni | Hacker News Reader

I don't get it. In their example of fibo(15), they already wrote the fibo function but how do they know it is correct ? Ok it is nice to have an autofill for expected value but not from the same 'thing' that I want to test...

View on HN

I'm not the OP but I think Fibonacci is a contrived example. In practice these "characterisation tests" are great for adding tests to an existing codebase. The "snapshot" just records the current behaviour; it doesn't judge whether it's correct or not, but if you refactor it will catch changes to the behaviour, and then it's up to a human to judge if the changes were desirable or not.

P.S. They're also great for writing tests for new code, for many of the reasons described in the OP.

Fibo would be excellent for property testing.

I.e. For all x, Fib(x+2)=Fib(x)+Fib(x+1)

A property based test produces values of x and tests this property (or invariant).

If it fails the framework usually then tries to find the simplest failing case.

I am always slightly uncomfortable about property based tests that just replicate the logic of the thing they're testing. I don't really know of an alternative in this case, and obviously there's rarely any harm in having regression tests. But it feels like a code smell somehow.

I suppose they're ideal for cases where there's a simple, obviously correct but slow implementation that you can use to test a more complex but faster implementation.

You say it replicates the logic, but I think it's more useful to say that it describes the meaning: fib is the function on natural numbers that has that property plus the base cases fib(0) = 1 and fib(1) = 1. That's what fib means.

That said, there are plenty of cases where your reaction is justified, where the test is implemented the same way as the function it's testing when the correctness property could be specified more simply.

You can chuck some regular tests in at the same time. They attack different classes of bugs.

Property based tests fuzz test values you may not have dreamed of. Given a string as the required type it could generate some exotic stuff.

Your uneasiness is understandable, but often it is far simpler to check the correctness of a result than to find it in the first place. Then property-based testing really shines.

Even outside that special case it is always possible to generate just some input data and check for the absence of runtime errors or exceptions. That approach can be improved by adding assertions, pre- and postconditions to your code under test.

Additionally, there a different ways to come up with good properties, see: https://www.youtube.com/watch?v=zvRAyq5wj38

Fib(x)=0 does satisfy this property though.

A full definition would include Fib(0) = 0, Fib(1) = 1 or if you prefer 1, 1.

You need to manually check the first 2 values in a seperate test.

I think what they've done is implemented a way of doing snapshot testing by putting the snapshot data in the test itself. In snapshot testing you just cache the result of a code block and use that cached data to check the code produces the same output as it did when you knew it worked. Putting the snapshot inline in the test makes it a lot easier to read.

IMO The real value in tests is making sure you don’t break things later, rather than for getting your current code working.

Maintaining tests that only test that things haven't changed can be a nightmare if you actually want things to change (like when you're making a change to the logic of a system). They can also be a real pain to maintain since if they fail you almost never know why they failed, just that they failed. This gives you (or the future maintainer) very little help in figuring out what's going on.

Guidelines for tests that reduce maintenance burden rather than add to it:

* Test the externally visible useful properties of your system, rather than implementation details. Related: don’t bother with mocks unless there’s no other way

* Keep your tests DRY: if there’s some invariant you want to check have it appear once in your test code

* Don’t over-test: you can delete tests and checks that fully overlap with other tests

* If you need to over-test make it easy to regenerate the expected results from your implementation (characterisation tests) as espoused by the article. * the amount of testing effort should be proportional to how important the functionality is. Reduce the effort for low impact features

That’s a big assumption. In TDD you write the tests first, to help you achieve the results expected, this doesn’t fulfill that need. It is basically snapshot testing applied to any kind of function.

I am of the opinion that the true value of TDD is that tests get written at all. I suspect that in practice the tests tend to get written at roughly the same time as the code, rather than strictly before.

IMHO, this is really more a documentation of the original behavior and maybe a test for consistency of later implementations against the original one. But it's not about validity. I'd call it a "traditionality test".

Not fib, but... I've written tests like this semi-recently for an algorithm that I had to devise. I went to Excel to generate the test data, based on a data pull from our production database. So, real production data, using a sequence of Excel spreadsheet functions, which also has multiple eyes on it (including legal, since this was regulatory in nature), fed as input to the algorithm to verify the output matched up.

I'm still on the fence about that but I'm not sure how else to test something that is essentially a glorified algebra formula.

Sometimes verifying a result is correct is easier than coming up with the result yourself.

Reply on news.ycombinator.com