Where can I learn about techniques like that? Any recommendation would be very appreciated!
Make sure to check out the references. I especially like "Ertl, M. A. and D. Gregg, The structure and performance of efficient interpreters, Journal of Instruction-Level Parallelism 5 (2003)"
Do you know of any recent efforts to combine ML and VM optimizations ?
(I read about the VS compiler and profile-based optimizations)
Though you might not think of hill climbing as "machine learning".
the repo and performance measurements are on Github[1]. It includes a couple of dispatch techniques, register caching, etc.
http://realityforge.org/code/virtual-machines/2011/05/19/int...