I'm confused by the idea that there could be a vulnerability while reading RTFs, of all things. Annoying; I frequently send out documents in RTF format to make sure everyone can read them...
Typical example of a buffer overflow that leads to execution specific to this case, although I'm sure this has been posted a million times here:
1. RTF parser allocates char[15] on the stack for storing the font family from {\fonttbl\...
2. Joe hacker shifts out a font definition of 15 chars plus a return address to some arbitrary code and the code itself.
3. The parser reads until the end of the fonttbl definition ;} which is longer than the buffer specified.
4. This blindly overwrites the return address by writing beyond the end of the buffer and pokes some arbitrary code on the end. If you're lucky this won't cause a bus error / MMU trap
5. The function returns and the malicious return address is jumped to executing Joe Hackers code.
Ultimately anyone can make this mistake when programming in C/C++ and they frequently do. It takes considerable skill to not blow your toes off like this.
There are certain mitigation method which prevent this but they aren't universally enabled or available or 100% effective.
I've pretty much given up on C these days but I spent a lot of time making sure stuff like this was impossible and most importantly avoided parsing anything myself when nice tools like lex/yaxx are around that have all this stuff safely wrapped up. I didn't win every time.
However, that might be bad practice.