First, most programs do much of their work inside the OS - rendering, network, interaction, whatever, so that's not emulated, Rosetta just calls the native OS functions after doing whatever input translation is necessary. So, nothing below a certain set of API's is translated.
You have to keep a separate translated binary in memory, and be able to compile missing bits as you encounter them, while remembering all your offset adjustments. It worked amazingly well during the PowerPC transition. Due to so many things running natively on x86, the translated apps frequently faster than running native on PowerPC macs!