The Design and Use of QuickCheck

Can you please provide the link to the babel plugin which creates the random values from flow types?

It's not one or the other, IMO. They're both useful tools. Unit tests for the bulk of my testing, and then property tests to do some light fuzzing against properties I want to hold.

I've found quickcheck fantastic when building anything that takes user input. It will find some amazingly specific strings that break that one library you use never thought to test, or code that unexpectedly shells out.

But it isn't the answer to everything- sometimes you really want to test just one case.

I've done a fair bit of testing with ScalaCheck. I do think it's useful to define your "generators" for your domain model(s), because without them, unit tests can end up being mostly data creation code. With your generators out in their own (testing) package, and named appropriately, you can get quite a bit of mileage out of them, even if you don't take advantage of the property-based testing framework beyond that. Something like..

    object Generators {

      val commentsGen: Gen[Comment] = ???
      val userWithTwoCommentsGen: Gen[User] = Gen.listOfN(2)(commentsGen).map(comments => new User(comments))

      val userWithNoCommentsGen: Gen[User] = Gen.const(new User(List.empty)
    
      val userGen: Gen[User] = Gen.listOf(comments).map(comments => new User(comments))
    }

And then your test code can get right to the point so to speak

    "Users with comments" should "karma equal to the sum of their posts" in {
      val users = Generators.userGen.take(10).foreach(u =>
        u.karma should be (u.comments.map(_.points).sum)
      )
    }

Some property testing tools like Python's Hypothesis[1] allow specifying specific example values for properties in addition to a general set of values so you get some specific deterministic tests. Hypothesis also saves falsifying values it found previously in a database that it reads from the next time you run it.

[1] http://hypothesis.works/

I use ScalaCheck all the time. I love defining my tests in terms of properties, and in some cases have a real, mathematic proof that my code does what I want (only possible when the domain of possible values is small enough, that's true).

One thing to remember when using an assertion-based method is what it can be used for:

1. Unambiguous specifications [vs human language] of exactly what the function expects, should maintain, and should output. This is basis of Design-by-Contract method of verification in Eiffel (original), Ada, and SPARK.

2. Ability to check inconsistencies on that if a formal, specification tool. An example is where unique types are defined for miles and kilometers that require a conversion before used in same-sized variable. Might have prevented a rocket from being destroyed.

3. Ability to formally prove through automated or interactive means that your code has those properties in all cases rather than whatever you tested for or your fuzzer came up with. Frama-C, Java w/ JML, and SPARK Ada can do this with SPARK the champ so far. It's hard and limits how you express the problem or what problems it can handle but with highest payoff in correctness. Anything unproven can be handled by a runtime check the tool might even insert for you.

4. Academic and commercial tools exist that can automatically generate tests from those specs. So, you get tests covering whatever you can specify automatically with no human effort. You can use your brain on stuff the tools can't handle.

5. Ability to do equivalence checks more easily on annotated (i.e. constrained) programs can help show optimizations or compilations didn't break your program.

6. The annotations might also be used for synthesizing programs or optimizing them even more than usual.

So, these are some benefits that come from leaning toward formal specs of code or within code instead of just manual, unit testing.

I never really understood the arguments against "non-deterministic" testing. A property based test will check more cases, and therefore detect more bugs, than the corresponding hand-chosen unit tests (in which you probably forgot at least 2 of the important corner cases).

All determinism guarantees is that you'll either reliably detect a bug or reliably not detect a bug, and hence get a nice aesthetic line of green checkmarks followed by red crosses in your test history. Or just get a nice looking line of green and not even know the bug is there. This seems like a small thing to sacrifice to detect more bugs.

Of course, regular unit tests have their place too, when your specification can't really be expressed as a "forall" property.

I use QuickCheck for all of my testing whenever I'm using Haskell (along with tasty-quickcheck). Note that property testing can also be used for unit testing; just don't use any variables in the properties :)

The rule seems to be that about 1/3 of the problems it finds are legitimate bugs in the codebase, 1/3 are problems with the property (e.g. stating something a bit too strong), and 1/3 are problems with the data generators (e.g. not satisfying some invariant).

The latter 2/3 can either be seen as overhead, or as fixing bugs in one's understanding of the codebase. Usually it's somewhere in between.

I don't see it as a 'conversion' per se, but I've been doing something similar to property checking for years in various tests, and therefore I don't think this is necessarily a new 'discovery' but more of a refinement of a codification of an existing, long-standing (decades) practice.

We've used it in at least one internal project. I found it to lessen readability and make maintenance harder. It could be the way things were set up. I like the basic idea, but that lib just wasn't doing it for me. We got close to something usable with test case generators where you replaced the intended bad value(s). There ended up being too much indirection for my taste for too little reward.

Actually, that 20 number probably relates to oddball uses of it outside of *_test.go files. Adoption in tests is likely decent after all.