I Hate It When My Tests Pass

blog.amir.rachum.com

9 points·Garbage··8 comments

This is why you should practice test driven development. If you write the tests before you implement the solution then you will know that your tests are doing what they are supposed to do.

Am I the only one who purposefully design my tests to fail on the first run to verify that they're doing what they're suppose to?

When you run the Django test runner with default verbosity, it shows a dot for each test so you see it's progressing and actually running the tests.

It has the problem though, that even with the lowest verbosity setting, it still prints out the summary of the tests, so every night I get email from cron confirming that the N tests still pass..

Maybe there's a place for some sort of a smart test runner that would execute assertions multiple times with random input, trying to achieve failure. If all attempts succeed then it'd mark test as questionable.

Have you heard of fuzzing?[1]

1. http://en.wikipedia.org/wiki/Fuzz_testing

I haven't, thanks for pointing out. Seems like a closely related concept, although appears to be used mostly in security testing context. Also, at least for Python most tools are a bit too complex—I imagined something more like a TestCase subclass (not even a test runner) that augments certain assert methods and possibly provides new ones. Anyway, on a second thought, adopting TDD looks like a better solution.

There's this: http://ruby.sadi.st/Heckle.html

It changes your class, then runs your tests to make sure they fail. I haven't tried it myself.

Instead of changing the asserts in the test, change the code you're testing. Inject an intentional failure and verify that your test fails.

Reply on news.ycombinator.com