Hacker News Reader

Comment by cortesoft | Hacker News Reader

If you mean randomized tests as in adding random choices to your tests, I think that is a very bad idea (that I have been bitten by before).

You NEED tests to behave consistently. If you run the tests twice against the same code, it needs to return the same result. Otherwise, how will you know you have fixed the issue the tests caught?

Speaking from experience, soon after you introduce randomness to tests, you will start to get intermittent test failures. When that happens, people will start to ignore failing tests because, 'hey, this is probably just a random failure'

View on HN

It's possible to have both randomness and reproduciblility - generate a random seed, log it, then explicitly seed the RNG. If you encounter a failure you suspect is caused by a particular random seed, temporarily hard-code that seed while you debug it.

The "that's an intermittent failure, just re-run it and it usually passes" attitude is in my experience more likely due to just plain old poorly-written tests, usually with a time.sleep() or something similar that makes them unreliable.

Seeded RNG.

Exactly, you use a fixed RNG seed so that the behavior is reproducible.

For complex multithreaded code, it can become difficult to achieve consistent execution, but a fixed seed is at least a good start.

So not really random, then? What is the point of adding an RNG if it is going to output the same values every time?

Because then the testing stimulus is not hard coded anymore, but generated. It's then trivial to run longer sequences, or tweak the test.

In hardware design, where reliability is key (making a chip costs multiple millions and 6 months time), there is only random testing (with seed for reproducibility), not "that bunch of case that the person testing thought of that day"

If you generate 10k test cases that find edge cases you wouldn't have normally found, that's value. If the framework is deterministic and always runs the same 10k test cases, that's generally valuable as well.

I'm not aware of anyone currently looking to ensure their test cases our generated using cryptographically random data.

Reply on news.ycombinator.com