AFLFast vs. Erlang
mindscope.pw
mindscope.pw
...
> Lucky for us there is a trick we can use to greatly improve that speed.
I just dusted off my fuzzpy project [1] this week and was bringing it up to date with recent CPython, LLVM/libFuzzer.
> Notice the speed: 373.4/sec, up from 20! And we've already found 100 unique crashes!
Most of my test cases run at < 10/sec. I feel like I just got lapped while I thought I was on a casual solo stroll. O.o
I intentionally left Py_InitializeEx() in the critical section for fear of contaminating global state, though. But I see that I should've at least experimented with it out.
[1] https://bitbucket.org/ebadf/fuzzpy
EDIT: sorry, I went a little off the deep end -- this post is in response to the article listed in the first paragraph "After finding 1800+ crashes in Python 3.5 using a method shown here ... "
PyObject *arg = PyTuple_GET_ITEM(varnames, j);
> ... can be null if varnames is an empty tuple (which it can be if the pyc file is malformed).This might be a case of "program works as designed" if the dev team thinks that's an acceptable reaction to malformed pyc input. But, hey, it's worth reporting because I would probably be inclined to fix it if I were the BDFL.
The OTP team is usually pretty good at working with tooling for correctness around the VM, so I would think this isn't too hard to get going.
AFL fuzz is severely limited if you need to boot up the VM, then run a program, then stop the VM again. It is rarely that part you want to test.
Perhaps we can strike middle-ground. Rip out the ETERM encoder/decoder to binary data and fuzz that. It should be a far simpler target.
And I suspect Erlang's VM is more complex. Has a register VM machine. A M:N scheduler (M cpus, N processes). Fairly intricate memory allocation, hot code loading and such.
(normal afl) http://mindscope.pw/static/afl-python-slow.png
(aflfast) http://mindscope.pw/static/afl-python-fast.png
sorry for screenshots ;)
QuickCheck uses hand-crafted known properties and generators. Fuzzing don't. QuickCheck can be used to find negative tests, but this is the sole target of a fuzzer. There are some similarities in the approach, but it is not the same kind of tool.
[1] see big red warning box at https://docs.python.org/3/library/marshal.html
We did similar excercise a year ago with zpaq, and Matt and community was able to fix it so well that now you can fuzz it pretty much forever :)