evmunro

Great idea to make it just a simple URL change. Reminds me of the youtube download websites.

I made a similar CLI tool[0] with the added feature that you can pass `--outline` and it'll omit function bodies (while leaving their signatures). I've found it works really well for giving a high-level overview of huge repos.

You can then progressively expand specific functions as the LLM needs to see their implementation, without bloating up your context window.

[0] https://github.com/everestmz/llmcat

You're right, it's likely that almost nobody is using `fmt.Sprintf` to build SQL queries in production.

Templating and `fmt.Sprintf` are essentially the same thing in this context - `Sprintf` just gets the point across in fewer lines of code, and allows people to come up with realistic scenarios themselves.

This is a great resource on fuzz testing algorithms & internals. I find myself coming back to it occasionally when building new fuzzing techniques at Fuzzbuzz.

OP and Founder of Fuzzbuzz here - let me know if you have any questions about fuzz testing, especially any particularly tricky cases you’ve run into.

I agree - I took a look at the minimization algorithm[0] and it seems like it loops through a few basic options, with the last one basically normalizing all possible bytes to something readable (like "0"). Part of the issue with trying to be as generic as possible is you sometimes can't find the best solution to every problem, this might be one of those situations.

I know the goal of 1.18 was to get the UX down, so I'm interested to see how it improves for 1.19.

[0] https://github.com/golang/go/blob/master/src/internal/fuzz/m...

I noticed that as well - most fuzzers will have a maximum duration or number of iterations they're allowed to attempt when minimizing so as not to starve out actual inputs. It could be that the fuzzer hit that limit, or potentially prioritizes readable inputs over small inputs.

Thanks for the questions & feedback! Concise docs are really important so this is all super useful. To answer your questions one by one:

1) The BrokenMethods are simple examples of programs that crash on buffer overflows/index out of range errors. If you were to pass "FUZ" into the Go method, it would check Data[3], thus causing a panic since there are only 3 elements in the string.

ninja edit: that python method in your comment IS a valid method with no error - a bit of a brain fart on my end when writing out the docs. It's been changed :)

2) In general a failure is any non-zero exit. We do this to be flexible in the way you report bugs. For C/C++ and Python this is usually with assertions, and in Go you can achieve something similar with:

  if !x {
    panic("Error")
  }

We also have other checkers or "sanitizers" that run with your code to look for certain bugs. For C and C++ code we support tools like Address Sanitizer, which report memory bugs like Heap Buffer Overflows and UAFs, and for Golang you can choose to fuzz your code with a race condition checker. These are just some of the examples of more advanced fuzzing methods we support, and we'll be making nicer tutorials/screencasts to showcase those over the coming week.

3) Thanks for the fixes - much appreciated. And yeah, we know GitBook is pretty slow, and we're in the process of moving to another docs provider.

If you've got any more questions please let me know!

Sure! Some of the classes of bugs that remain low-hanging fruit for languages like Python include slowness, hangs, panics, race conditions, assert failures, excessive resource consumption and other Denial of Service attacks.

Other use cases include using fuzzing to compare implementations of libs that require the same functionality, detecting invariant violations, testing implementations that are meant to work together (i.e. serialize(deserialize(x)) == x).

In general fuzzing C/C++ libraries for memory bugs is the most commonly described use-case, but I think there are tons of fuzzing use cases that haven't been thoroughly explored yet.

Radamsa is awesome! Definitely agree, and one of the goals for Fuzzbuzz is to be able to hot-swap between fuzzing backends without any interface changes (or to use all backends at the same time, to account for differences in findings).

re: pricing, we do offer infinite scalability in terms of CPUs, but that might not be as clear as we'd like it from our pricing page. Or maybe I'm misunderstanding you. Either way, if you have any more thoughts/suggestions on pricing I'd love to hear it.

Thanks for the link! We've been looking at all the current AFL-like/AFL wrappers for Java as we decide how best to implement Java fuzzing in Fuzzbuzz, and yours looks pretty nice.

Definitely going to play around with this :)

Memory security issues have been the main focus of fuzzing, but it's really useful for other use cases as well, such as: slowness/hangs, assert failures, panics, excessive resource consumption and DOS attacks. We've also done some work with Go to detect race conditions while fuzzing.

You can also do differential fuzzing to compare 2 different implementations that solve the same problem, or fuzz for invariant violations/assertion failures. I think the possibilities extend far beyond just memory safety, and I'm really looking forward to finding other areas in which fuzzing is applicable.

We have! afl.rs[1] is awesome, and seeing as it's found some interesting bugs, I think Rust would be a great addition to Fuzzbuzz. It's on our roadmap.

[1] https://github.com/rust-fuzz/afl.rs

Yep, we're definitely going to integrate more automated analysis. As of now we do some rudimentary analysis based off the type of the bug (Heap buffer overflow, UAF), read/write size, and similar metrics, but we'll be adding more advanced methods of categorization as the platform matures.

We've been thinking about the best way to use Fuzzbuzz to benefit the OSS/bug hunting community, and the integration idea is a great one. We're also providing free plans with extra CPU power for security researchers & bounty hunters.

Thanks for mentioning us on the issue! I'd love to help get that project up and fuzzing

We actually distribute the fuzzing workload across physical machines, for precisely that reason. Each instance of AFL gets its own kernel & physical core, and we use a staged synchronization algorithm to make sure all of the machines' corpuses stay up to date.

All of this was done to try and keep the scaling as linear as possible, so that when you double your CPU count you're doubling your execs/second as well.

If you're interested in giving it a go, I could set you up with an OSS plan & some free CPU power - let me know!

everest@fuzzbuzz.io

Really interesting to see the desire for ruby support in this thread! It's definitely on our roadmap.

Shoot me an email at everest@fuzzbuzz.io and I'll let you know when we launch ruby fuzzing.

They certainly could if their project is large enough! Every widely-used C/C++ project should use OSS-Fuzz, it's an awesome service.

We support a couple of languages that OSS-Fuzz doesn't (Go & Python as of now), which is why I thought this was worth mentioning :)

We're already sort of integrated with GitHub, since you can integrate your projects to automatically pull updates from repositories, so GitHub login is definitely on the roadmap!

Since the type of fuzzing you can do right now on Fuzzbuzz is language-specific, you wouldn't be able to fuzz Javascript code.

We are in the process of building a fuzzer for generic web-apps, so watch this space :)

That's a problem that we've been thinking about a lot. The way our fuzzing works right now is that your method consumes an array of bytes, which you can then use to build up arbitrary structures. It's simple, but manages to be really generic and flexible at the same time. Of course, it means you do have to define what your inputs look like.

We do have plans to build some tools to make this easier. I'd like to see a scenario where defining inputs is as simple as specifying the data types that your code requires. (Or perhaps even automatic detection, for less complex cases)

We don't support protocol fuzzing yet, but it's definitely on our roadmap - we wanted to start in an area that we felt was lacking the most, and then move into other types of fuzzing. We do have some novel REST API fuzzing techniques in mind that we're really looking forward to implementing as well.

We're also thinking about extending to property-based testing. There are some really awesome hybrid testing tools out there, such as DeepState (https://github.com/trailofbits/deepstate) which combines Symbolic Execution and Fuzzing behind one clean interface, and we'd really like to push the boundaries of that type of testing.

And yep, we do! Everything is written in Go, which is part of the reason it's one of the first languages we support.

Ruby is one of the languages on the roadmap, but it's not super high-priority right now, mostly because we haven't seen a lot of interest. If you have a specific project/use case in mind I'd be interested to hear more about it.

P.s. Know someone who maintains an open-source project, written in C, C++, Go or Python, that should be fuzzed? Please send an email to oss@fuzzbuzz.io. We’d love to fuzz your code for free on our platform and make the world’s open-source software more secure.

Either libFuzzer or AFL are your best bet for getting started - they both use very similar algorithms and just differ on execution.

libFuzzer is more suited to fuzzing a single method, while AFL gives you a little more freedom when deciding how to fuzz your code.

This is a nice initial look at libFuzzer: https://github.com/google/fuzzer-test-suite/blob/master/tuto...

And here are a couple of my favourite AFL tutorials:

- https://fuzzing-project.org/tutorial3.html

- https://github.com/ThalesIgnite/afl-training

Happy to answer any questions!

Fuzzing is super powerful, but can be a bit complicated to set up - that's why I'm working on a fuzzing-as-a-service platform[1] that automates a bunch of the steps described here.

If you're interested in trying out fuzzing without having to learn the intricacies of AFL or set things up manually, let me know[2] and I can get you set up with an account to play around with.

Happy to answer any questions about AFL/Fuzzbuzz!

[1] https://fuzzbuzz.io

[2] everest [at] fuzzbuzz [dot] io