Hacker News Reader

Comment by hexaga | Hacker News Reader

IME, random data is extremely useful. Consider fuzzing, property testing ala quickcheck, automated fault injections, etc.

The key thing to make it work well is to keep track of your prng seeds to make things deterministic. If you find that seed `484382943` makes a test fail, append it to a list of seeds to always re-test.

View on HN

It only works until the pattern of randomness consumption changes even the tiniest bit. But it's a valuable tool nonetheless, even if only for quick re-runs during a debug/fix session.

For long term regression checks you might want to make sure that your test data object graph can be round-tripped through JSON (or any similar format, but JSON tooling is ubiquitous) to get your random discoveries into a less fragile form (or: fragile in a different, more fixable way).

My thought would be that, instead of re-using the seed in further tests, you should be figuring out what inputs that seed generated and create a test case for those inputs directly.

Are random test cases only for checking code throws (no) exceptions? If not, how do you create the expected results from the random data without reproducing the method you call to begin with?

They only idea I get is to fix the seeds/inputs, write the expected results into a file (making it persistent), and checking them manually the first time. Subsequent test runs check whether the output changed.

You can test properties of the result without an assertion of the exact result:

Take adding two integers, you can test 3+18= 21 or you can assert that, when both arguments are positive, the result is greater than each argument.

No offense, but that's a toy problem that's hard to generalize to actual issues we see in our code. This might apply to some financial code where some complex situation should always result in a great or smaller result, but what about the things people typically work on? I'd wager a significant percentage of us are writing CRUD apps; what's the crudsmith's equivalent to this?

    expect(add(A, B).toBeGreaterThan(A);
    expect(add(A, B).toBeGreaterThan(B);

It works the same way with actual (non-contrived) problems. You'd identify a property you want to test, and assert that it holds for randomized inputs.

Say you have a CRUD api (TheTrackerOfWidgets), and you want to test a create Widget call. Widgets can have a name, size, and each is associated with a unique id.

Your property test might look like (in pseudo-rust):

    #[quickcheck]
    fn created_widgets_exist(name: String, size: u32) -> bool {
        let resp = create_widget(&name, size);
        if !resp.is_valid_json_or_whatever() {
            return false;
        }

        let id = resp.get_id();

        let resp = retrieve_widget(id);
        if !resp.is_valid_json_or_whatever() {
            return false;
        }

        resp.get_name() == name && resp.get_size() == size
    }

You would probably go further and not use default types but instead use bounded types based on any domain specific verification logic you have. Note that nowhere does it actually assert that a response looks exactly like what you expect, you assert that it is valid and/or satisfies some property (that created widgets exist and are accessible by further api calls).

Ahh! So with random inputs(from a pre-defined range) you can assert about properties of the result (if you find some), because you limited your input range.

Exact results still have to be created manually or automatically (randomly) created and the result manually checked once.

One test I've done a few times is a "round-trip test" with randomized data, to ensure that things can go in the system and come back out again the same. This could even be done all the way from the browser with selenium or whatever. Even something this simple, you'd be surprised what this can turn up. (I actually write "round trip" tests a lot, though not always randomized.)

For a CRUD app, I've also created random account generators that can randomly create all different sorts of accounts, based on the database layout. You can then even do things as simple as "render the view screen on this user account", and, again, you might be surprised what pops out. Plus, as you continue building the system, you can both extend your "random account generator" to generate more and more random things, and use it on more and more of your code. This is when you can discover that your page crashes if they don't have exactly the same number of shipping addresses and phone numbers, or your forum crashes on users who didn't enter the optional email contact address, or whatever. Also, as you have the ability to easily generate "100 fairly random accounts", you'll find other interesting tests that you can write.

Simple number comparisons are just an easy example.

If your CRUD app happens to extend just a wee bit beyond the boringest possible CRUD and you also have any sort of structured test you need to accept, you can also use a fuzz tool like http://lcamtuf.coredump.cx/afl/ You can find all sorts of interesting things in any sort of structured text input that way quite easily. (Theoretically, if you're motivated, you can even hook up a fuzzer to the random account creator idea above, but that does take a bit of work, but that would do interesting things if you could get it going. I've only used something like afl a few times, but it's kinda fun, and amazing.)

Reply on news.ycombinator.com