I'd start with reading the libfuzzer docs[0] and then read the syzkaller papers[1]

[0] https://llvm.org/docs/LibFuzzer.html [1] https://github.com/google/syzkaller

https://github.com/google/syzkaller