Hacker News Reader

Comment by staticassertion | Hacker News Reader

Great to see fuzzing becoming more mainstream. Ultimately we have absurd program states, with even a trivial program's state vastly exceeding the number of particles in the universe. We need to start finding order-of-magnitude-better approaches for testing.

I almost always write generated tests at this point with unit tests being a fallback for slow code or niche cases. What I dont generally write though is fuzz tests, which would really be a 'next step'. In Rust it's not very hard to do so, but it hasn't quite hit the "trivial" mark yet for me, whereas quickcheck is virtually the same amount of work to use as to not use.

Languages like Go adopting and mainstreaming these practices will be a benefit to everyone.

I'm curious if there's documentation on:

a) The coverage approach taken

b) The mutation approach taken

Can you configure these? Plugin different fuzzing backends?

View on HN

> Great to see fuzzing becoming more mainstream.

Agreed. I wrote a fuzzer at my last job and it found a bunch of bugs right before a release. Nobody knew what fuzzing was so I was attacked by the program owner for trying to break the software and given an insulting performance review for it. Then I had all the fuzzing results and coredumps deleted out of their directories by the program owner so the release looked immaculate. Defense software ftw

Yikes, sounds pretty toxic on their part, but good on you for taking a strong approach to software stability.

Also, writing fuzzers is super fun.

In such projects there is often a promised release date and a contract stipulating e.g. a total test pass rate of at least, say, 95% or something like that.

Now if you use a fuzzer to generate a lot of test cases that fail (if it only saves the failing ones) it will impair the ability to release.

So sadly there are often incentives to not fuzz close to a release.

The management euphemism is “taking a risk based testing approach”, when selectively removing test cases to make the numbers to qualify for a release.

> I almost always write generated tests at this point with unit tests being a fallback for slow code or niche cases. What I dont generally write though is fuzz tests, which would really be a 'next step'. In Rust it's not very hard to do so, but it hasn't quite hit the "trivial" mark yet for me, whereas quickcheck is virtually the same amount of work to use as to not use.

Did you mean generative tests? You're talking about quickcheck and that's what it does.

"Generated tests" would usually be interpreted as codegen'd test which you commit.

Tests with generated input. Call it what you like.

What kind of generated tests are you writing?

Is it more similar to 'golden files'? Generate expected output and assert versus current implementation output?

Just taking what would normally be a unit test and having the input values be generated. Some examples:

1. I have a test for encryption/decryption functions. The data that's provided for the plaintext, additional data, and key, is generated. The assertions are:

assert_ne!(plaintext, encrypted_data);

assert_eq!(plaintext, decrypted_data);

assert_eq!(aad, decrypted_aad);

etc

2. I have some generated integration tests. For example, in our product, there are certain properties that should always hold for a given database entry. I generate a new entry on every test and have the fields for that entry provided by quickcheck, then I perform the operation, query the database, and assert that properties on those values hold.

So to answer your question, yes. Sometimes you want to check a concrete output (ie: "this base64 encoded string should always equal this other value) for sanity, but in general property tests give me more confidence.

I find it works particularly well with a 'given, when, then' approach, personally.

edit: I'll also note that for the base64 case I'd suggest:

a) A hardcoded suite of values.

b) Generate property tests.

assert_eq!(value, base64decode(base64encode(value));

As well as things like "contains only these characters" and "ends with [=a-zA-Z]" etc.

c) Oracle tests against a "known good" implementation.

Sounds like a sensible mix. There is really no single silver bullet.

We at https://symflower.com/ are working on a product to generate unit tests. Unlike quickcheck/proptest we promise to find errors, even if they are unlikely (for example [this input](https://github.com/AltSysrq/proptest/blob/master/proptest/RE...) would be trivial for Symflower). Also, unlike fuzzing our technology is deterministic.

Here's one of our blog posts that explains the approach: https://symflower.com/en/company/blog/2021/symflower-finds-m...

> we promise to find errors

What does this actually mean, because I assume you didn't disprove Rice's theorem?

I don't think Rice's theorem being proven invalid (or rather, more likely (but imo unlikely), P=NP) is important to this. In fact, a somewhat loose interpretation of Rice's theorem would imply that you can not "prove your way" out of having bugs of arbitrary classes, which seems entirely compatible with "our testing framework will find bugs".

Rice's theorem is applicable; the paradox is the usual trivial case,

    if symflowerSaysItsBroken() {
        DoTheRightThing()
    } else {
        DoTheBrokeThing()
    }

I have no doubt that symbolic analysis can find a large class of problems, but if you write stuff like "we promise to find errors" you will a) dupe a lot of junior devs who will genuinely believe your tool is doing impossible magic, b) alienate experienced devs who know that's 100% marketing fluff and find nothing more specific on your site.

I'm pretty much the ideal customer for a product like this - staff+ with final say about toolchain decisions on a product with a lot of data-driven behavior. But what I want to see isn't a vague "promise" (really? is it in a contract, that you're liable for bugs your tool misses? of course not) but some actual comparison to quickcheck/fuzzing. I want to see your approach finds a superset or at least mostly-disjoint set of what we already have invested in, or finds the same set more effectively. Like, survey through the bugs fuzzing found in go stdlib and show me your tool finds them all faster.

I'm also skeptical of some "purely" coverage-driven approach from the start - coverage-driven fuzzing already has a tendency to miss interesting cases that don't come from branchiness - a classic example is subnormal value handling. Fuzzing usually still finds these eventually just by virtue of exhaustiveness; a tool driven only by symbolic methods better come armed with a _lot_ of encoded knowledge about the language semantics.

From looking at their stuff I think their value comes not from the the way they promise to write tests that exercises more code paths than human generated testing can. This is different from the Go fuzz testing approach, which requires you to write a specialized kind of test by hand. I can even imagine a future evolution of Symflower's product that writes fuzz tests for you.

Go stdlib has property testing built it. It's not as powerful as some quick check frameworks, but it's built right in. I wrote an article on it.

https://earthly.dev/blog/property-based-testing/

If state space is your concern, I would try proofs. Here’s a live demo from Martin Kleppmann about proving something about a stateful distributed system: https://youtu.be/7w4KC6i9Yac.

The tools have come a very long way, Isabelle is quite usable after learning a few concepts.

I believe this is the order-of-magnitude better approach you’re thinking of. We can apply finite effort to a proof about an infinite state space, with no runtime cost.

And, Isabelle has great automation. As seen in the video I shared, proofs can often be found with a little nudging in the right direction. You don’t have to write the whole thing out.

Reply on news.ycombinator.com