I eventually changed my opinion into JIT being the only way to make dynamic languages faster, while strong typed ones can benefit from having both AOT/JIT for different kinds of deployment scenarios, and development workflows.
Which is not exclusively an argument for runtime JIT— it can also be an argument for instrumenting your runtime environment, and feeding that profiling data back to the compiler to help it make smarter decisions the next time. But that's definitely a more involved process than just baking it into the same JavaScript interpreter used by everyone— likely well worth it in the case of things like game engines, though.
In finance the performance critical code path is often the one run least often. That is you have a if(unlikely_condition) {run_time_sensitive_trade();}. In this case you need to tell the compiler to ensure the CPU will have a pipeline stall because of a branch misprediction most of the time to ensure the time that counts the pipeline doesn't stall.
The above is a rare corner case for sure, but it is one of those weird exceptions you always need to keep in mind when trying to make any blanket rule.
http://eel.is/c++draft/dcl.attr.likelihood
Prior to this it was also possible, but required compiler-specific macros like GCC's __builtin_expect, which is used all over the kernel:
But what likely/unlikely mechanism can do is that it can serve as a hint to the compiler which the compiler can then use to generate more optimal code layout. For example, if we provide the compiler with likely/unlikely hints in our code, compiler will try to use that hint to stitch together the more probable code paths first. In theory this approach should result with better utilization of CPU instruction cache and thus it may lead to better performance.
If you want to cause a pipeline stall, there's usually serializing instructions like cpuid/eieio.
Regardless, you can at least lay out code in a way that heavily favors one branch over the other, and potentially optimize for one branch (I.e. give one branch very expensive prelude/exit to make another branch very cheap). The last thing I’ve seen compilers greatly struggle with though…
Assuming either source of data is the single source of truth for all optimizations is a fallacy.
Use the right tool for the right job. Use all the tools if you can.
Static + LTO w/ PGO is pretty much the practical ideal. JITs don't offer much until you start adding dynamically loaded code where LTO just isn't possible anymore.
Most PGO for AOT scenarios suffers from tooling experience and not using data sets similar to production workflows.
So What language/runtime out there ships unoptimized bytecode, an optimized precompiled static + LTO w/ PGO, and can re-optimize with runtime gathered information via a JIT?
Heck, what language/runtime is even designed around being performance-focused with a JIT to deliver even more performance in the first place? Pretty much ever JIT'd language makes trade-offs that sacrifice up-front performance and later hopes the JIT can claw some of it back. maybe this is kinda WASM-ish territory, although it then sacrifices up-front performance for security and hopes the JIT can claw it back.
As others have pointed out, HP's Dynamo managed to use runtime re-optimization to improve performance of many binaries without cooperation from the original compiler. Runtime optimization doesn't strictly require any of the information lost in optimized binary builds.
Last I checked, the Android Runtime would AoT-compile Dalvik bytecode at install time, and in the background re-optimize the binary based on profiling information. Though, I don't think it performs hot code replacement.
I'm not sure the latest with Oracle's Java AoT. Last I checked, Oracle's JVM wasn't able to inline or re-optimize AoT-compiled code through JIT'd code.
Some optimizations, such as loop unrolling make JIT'ing harder. However, strength reduction, loop-invariant code hoisting, etc. make the JIT's life easier. Back around 2005-2006, my employer was getting good mileage out of a Java bytecode optimizer. If your AoT and JIT are cooperating, the AoT can stuff any helpful metadata (type information, aliasing analysis, serialized control flow graph, etc.) into an auxiliary section of the binary.
I'd like to eventually write a C compiler that essentially compiles to old-school threaded code: arrays of pointers to basic blocks (strait-line code with a single entry point and one or more exit points). Function entry would just pass the array of basic blocks to a trampoline function that calls the first basic block. Each basic block would return and index into the function's array of basic blocks for the trampoline to call next. Function return would be signaled by a basic block returning -1 to the trampoline loop. A static single assignment representation of each extended basic block would be stashed in an auxiliary ELF section. On a regular system without the runtime optimizer, the only startup overhead would be due to bloated binary size. A wild guess at the performance overhead without the runtime optimizer would be in the 5% to 15% range. However, on a system with the ELF dynamic loader replaced with a runtime-optimizer, the runtime would set up a perf signal handler that would keep counters for identifying hotspots to trace. If the tracing conditions were met, the perf signal handler would walk back up the call stack to find the last occurrence of the address of the trampoline, and replace it with a version of the trampoline that in addition, stores the address of the next extended basic block to run. Once a trace loops back on itself or meets some other TBD criteria, the runtime would stitch together the SSA representations of the constituent extended basic blocks, and generate a new optimized basic block that's the inline of the components of the trace, and finally place the address of the new extended basic block in the correct place in the array of extended basic blocks, thus performing runtime code replacement. Anyway, that's my grand vision. I've taken the introductiory Stanford compilers course, and am slowly working my way forward, but I have a job and a young kid, so I'm not holding my breath.
Among other things, this allows for inlining of hot paths across dynamic library boundaries. It also improves code locality and should increase the percentage of not-taken branches in the hot path, which should help reduce problems with aliasing in the branch predictor.
Isn't low-latency trading only a subset of finance?
Once you've managed to debug the codegen anyway (see: The Long and Arduous Story of Noalias).
Naively, I would have assumed that branch information would be most valuable, in terms of being able to guide execution toward the hot path and maximize locality for the memory accesses occurring on the common branches. And that info is not something that would be assisted by more expressive types, I don't think.
This is where the profile guided optimisation comes in – for statically compiled languages, with a caveat being not always straightforward to come up with a set of inputs that will trigger an execution of all possible code paths. One solution is to provide the coverage specifically for the performance critical code paths and let the rest just be.
Aren't JITs already self-instrumenting? What would you instrument that the JIT is not already keeping track of?
For object graphs in Java you can waste exorbitant amounts of memory by having a lot of “children” members that are sized for a default of 10 entries but the normal case is 0-2. I once had to deoptimize code where someone tried to do this by hand and the number they picked was 6 (just over half of the default). So when the average jumped to 7, then the data structure ended up being 20% larger than the default behavior instead of 30% smaller as intended.
For a server workflow, having data structured tuned to larger pools of objects with more complex comparison operations can also be valuable, but I don’t want that kitchen sink stuff on mobile or in an embedded app.
I still think this is viable, but only if you are clever about gathering data. For instance the incremental increase in runtime for telemetry data is quite high on the happy path. But corner cases are already expensive, so telemetry adds only a few percent there instead of double digits.
The nonstarter for this ended up being that most collections APIs violate Liskov, so you almost need to write your own language to pick a decomposition that doesn’t. Variance semantics help a ton but they don’t quite fix LSP.
As for mobile, this isn't far off from the whole set of hand written Assembly interpreter, JIT with PGO data caching, AOT compilation on idle with the PGO collected data, sharing PGO metadata with other devices via the PlayStore, that Android happens to do since version 7, and refined later.
Most other languages only have mutable collections, and name collection types after their implementation details instead of what the user actually wants from them.