Please look at the test script again. It does indeed warm up the JIT.

The timing results come from within HHVM. Basically, the test is dispatched within HHVM, isolating just the function call itself. It's then executed 5 times (timed independently) and then the bottom 2 executions are thrown away. See: https://gist.github.com/ircmaxell/8859d7f1bbee7b2f6e16#file-...

So the warm-up period happens. It's not just "benchmark `hhvm file.php`". The benchmark happens inside of HHVM.

So yes, JIT warmup is accounted for 100%. In fact, PHP, HHVM and Recki-CT all execute with the exact same pre-compile steps (parsing and any pre-compiling are ignored in the benchmark).

And their AOT approach that they tried was all-or-nothing. Meaning the entire code base had to be converted. This is a "compile what you want, and run the rest normally". In fact, this is 100% compatible with Hack and HHVM...