For example LLVM's lib fuzzer uses instrumentation to track code coverage and mutates data to find invalid behaviour.
https://llvm.org/docs/LibFuzzer.html
It uses a compiler pass to insert code to branch points functions calls etc. I think it uses genetic algorithms to increase coverage by changing the data.
There are others that work in similar ways one of them is. https://github.com/google/AFL