There's a really interesting article from Tavis Ormandy about the instruction set and virtual machine used in RAR: https://blog.cmpxchg8b.com/2012/09/fun-with-constrained-prog....
The docs for the toolchain he implemented (https://github.com/taviso/rarvmtools) allude to a number of bugs, but doesn't sound (??) like they're related to this vulnerability.