Go Fuzz Testing | Hacker News Reader

Go Fuzz Testing

blog.fuzzbuzz.io

114 pointsandrei22 comments

That first example is deceptive. In this line:

    for i := 0; i <= n; i++ {

The "<=" character just looks like a "<" character due to the typeface that the author chose. I'm going to assume that wasn't an accident because it perfectly masks the obvious bug in the for loop.

When I did a quick scan of that code it looked good to me. Then he claimed it should panic and I felt crazy. So I copy/pasted the code into an editor to see wtf was going on and sure enough it was "<=".

This is my main issue with fonts made for programming that use special ligature marks. They _look_ cool, but they are very hard to tell apart from other characters.

It's like someone decided that the I, l, 1 confusion wasn't enough and decided to add more. :P

    go test fuzz v1
    string("000000000000000000000000000000Ö00000000000000000000000000000")
    rune('\u0083')
    int(60)

Interesting that the minimization engine wasn't able to shrink this further.

I noticed that as well - most fuzzers will have a maximum duration or number of iterations they're allowed to attempt when minimizing so as not to starve out actual inputs. It could be that the fuzzer hit that limit, or potentially prioritizes readable inputs over small inputs.

For string inputs, some form of binary search ("Check if the bug exists in the first half or second half of the string") would be able to reduce this example to "Ö" in only a few iterations. Not sure if this just isn't implemented, or whether there's something more complex going on.

There's also the fact that I'd expect a fuzzer that knows about Unicode and UTF-8 strings to have a known list of weird behavior hardcoded as seed values, and certainly two-byte runes would be on that list.

Of course, this is only the first release with the fuzzer, and it already looks really amazing - all I'm really saying here is that I can't wait for these to be features of the fuzzer in the future!

I agree - I took a look at the minimization algorithm[0] and it seems like it loops through a few basic options, with the last one basically normalizing all possible bytes to something readable (like "0"). Part of the issue with trying to be as generic as possible is you sometimes can't find the best solution to every problem, this might be one of those situations.

I know the goal of 1.18 was to get the UX down, so I'm interested to see how it improves for 1.19.

[0] https://github.com/golang/go/blob/master/src/internal/fuzz/m...

You might be interested in reading about delta debugging: https://en.m.wikipedia.org/wiki/Delta_debugging

The algorithm has similar complexity as binary search, but is a bit smarter on deciding how to split the test input at each iteration.

I’ve been studying this in my masters, and we’ve recently had to write a Java implementation. I’m keen to start on a Go package soon that might work well with fuzz testing.

It’s a bit surprising to have fuzzing as part of the standard tool chain, but not property based testing.

I guess we still need to rely on something like gopter.

Use table driven tests.

func TestFoo(t *testing.T) { list := []struct {

     want string
     input string
     expectedErr error
  }{
  {item1},
  {item2},
  }

  runtest

That’s not property testing. Yes there is a property being checked but you the user are selecting the inputs by hand which lumps in whatever biases you might bring to the test. Property testing is a property to validate, randomized search over the input space and, ideally, shrinking of inputs that fail the property. The benefit is you get to search over more of the input space that a human would normally do with an explicit table of inputs and with hopefully less bias toward inputs that don’t fail the property.

testing/quick is in the standard lib, though it's a very far cry from Hypothesis or QuickCheck.

> It’s a bit surprising to have fuzzing as part of the standard tool chain, but not property based testing.

Property testing doesn’t really benefit from (let alone need) toolchain integration, so while having better support for proptesting would be nice and give them more visibility, it’s far from critical.

I'm pretty sure fmt.Println("HELLO", str, len(str), len(result), n) isn't supposed to be in the final solution? :)

Good catch :) fixed!

Neat, Andrei. Good luck with this venture!

Curious if fuzzing is a common thing in network programming, does fuzzing plays nicely with binary inputs?

Along side file parsers it’s a pretty major fuzzing target, as it tends to be exposed to malicious inputs.

Fuzzing works primarily on binary data, “structured” fuzzing is somewhat rarer.

It does! Fuzzing actually started off as a tool built by security researchers to find vulnerabilities in parsers, and other complex codebases, usually written in C/C++ (looking for memory bugs). So anything that deals with untrusted binary data is a prime candidate for fuzz testing.

Go’s fuzzing framework supports `[]byte` arguments as well as all of the standard Go primitives, so you should be able to test netcode this way.

If you're looking for a C/C++ solution, my recommendation is libfuzzer [0]. We've also built our own C/C++ fuzzing engine at Fuzzbuzz [1].

[0] https://llvm.org/docs/LibFuzzer.html

[1] https://docs.fuzzbuzz.io/docs/getting-started-in-c-or-c++

What’s the rationale behind bundling a fuzzing library in the language instead of as a separate library?

Probably to encourage a standard fuzzing library for the language. Like the same way go fmt is part of the language.

It does instrumented fuzzing. The older https://github.com/dvyukov/go-fuzz would rewrite your sources to inject the instrumentation and pass the rewritten sources to the compiler, but it didn't really work with Go modules. This is something that probably makes sense to integrate with the compiler toolchain, same as `go test`'s coverage testing.

To do guided fuzzing (graybox, like afl or libfuzzer) you need to instrument the binary. To instrument the binary, you need to be part of the building process.

Since Go has a bespoke compilation toolchains and AFAIK doesn’t have compiler plugins, external fuzzing tools had to either fork the toolchain or perform extensive pre and post processing (couldn’t tell you what go-fuzz did but many article about go-fuzz note that the building process can take a while).

As such, building fuzzing into the standard toolchain and maintaining it as part of the project makes a lot of sense. It also gives fuzzing a much higher level of visibility (because sadly there will always be a population for whom an external / third-party tool will be suspicious).

Reply on news.ycombinator.com