Property-Based Testing in Rust with Arbitrary

Most posts about property-based testing focus on convincing the reader to do property-based tests and the mechanics of writing the tests.

Unfortunately, I'm already convinced and know how to write the tests — what I wish I could find is some resource that would hone my ability to identify worthwhile properties. The best one I know of and go back to time and time again is https://fsharpforfunandprofit.com/posts/property-based-testi..., but I really wish I could find more.

It looks like Arbitrary will sample integers from an uniform distribution. For property based testing you often would like to have a bias towards "interesting" values that might trigger special cases, like -1, 0 and 1.

E.g. if you had "max_speed_kph: Option<u64>", your tests might never check what happens if max_speed_kph is 0.

There are a couple of useful things missing here at a glance. One is shrinking, where the library will attempt to find simpler cases of failures rather than just brute force a random one. Second is how to create related pieces of data that are constrained in some way, above and beyond pure random bytes, but I assume there’s some way here to manually interpret the random bytes. But sometimes you have to hand-tune these things because you don’t just want to wait for the RNG to guess the right combination to create valid data.

I’ve had some success with property based testing, at least for heavily algorithmic code. For example I wrote a kNN implementation after finding all sorts of edge cases in others (floating point shenanigans included). You can succinctly capture properties like: returning the right number of clusters, each point being closest to its cluster’s mean, each cluster being centred on its centroid, all points being assigned to a cluster, and all clusters being non-empty. But even then you could game things if you wanted to.

Where I’ve struggled is in the general case. Most tests don’t have good property based alternatives, beyond just repeating the logic of the code under test, which seems pointless. And so you have this constant tension of wondering if a particular test could or should be property based.

The value in property based tests is that the original libraries (Quickchwck, both Haskell and Erlang) try to find the smallest reproducible error sample, and then display that sample to you.

And that (Quickcheck at least) has a bunch of primitive generators that you build your structures out of.

No idea, if it's doable in Rust. I just find it strange that it works on a stream of random bytes out of which you magically get some structures.

Meta question: do many people use property-based testing in their production systems? I think the philosophical idea is extremely sound and is a very good approximation to proving correctness, but I only ever see trivial examples like "a + b = b + a" as a property. How does property-based testing hold up in far larger systems with quite complex setup behaviors?

We use https://docs.rs/proptest/latest/proptest/ extensively at Remind. It very recently got a few new maintainers and does shrinking pretty well.

Our biggest use case has been testing our database materialization. We have a somewhat complicated hierarchical organization tree/forest within our database. In order to efficiently query those structures, we materialize the transitive closure of these relationships. TL;DR; we populate a row for each (org, ancestor) and maintain that via postgres triggers that fire whenever we mutate things.

The prop test that tests it all does three things. First, it establishes an arbitrary that captures all the possible shapes of organization trees. Second, we define the set of mutations (add a child, switch parents, delete a grandparent, etc.) that may occur. Third, we have code for reading/writing org trees to and from the DB.

The test then generates an org tree, syncs it to the db, performs 1 or more mutations on the tree and then compares what's in the database with the resulting tree.

Overall it's been pretty great. I highly recommend property tests, especially when modeling complex data in an external system.

> I don't want From to panic

There’s also a TryFrom which returns a Result, in the event your conversions are fallible.

Great name, cant wait to search for arbitrary rust /s

Just wondering, is there a reason for not having more specific names in rs projects?

I had a really hard time getting past the bad construction of the struct. Why not make an enum variant where a bike just doesn't have a fuel property. Makes things a lot easier.