I don't know if I'd call what libcap does "compiler theory" it appears to be a darker art that has extra knowledge about a lot of different things. It's remarkably sophisticated though. I implemented a BFF jit engine for a popular ips and spent a lot of time trying to work out a good way to replace libcap before we just went with it. Optimized libcap output will routinely make you scratch your head and wonder how it can work.