Writing memory safe JIT compilers
medium.com
medium.com
On small benchmarks I can believe the performance can be very similar between them, but I'd be more interested in large real-world codebases. I'm not aware of any myself, and can't seem to find any. Does anyone know?
this is a perpetually repeated misconception:
> Why did we Abandon Partial Evaluation?
https://www.pypy.org/posts/2018/09/the-first-15-years-of-pyp...
> others do
to my knowledge graal is the only production project using futamura projections (what you're talking about)
but that is not what PyPy does
> So, how did that tracing JIT generator work? A tracing JIT generates code by observing and logging the execution of the running program. This yields a straight-line trace of operations, which are then optimized and compiled into machine code. Of course most tracing systems mostly focus on tracing loops.
> As we discovered, it's actually quite simple to apply a tracing JIT to a generic interpreter, by not tracing the execution of the user program directly, but by instead tracing the execution of the interpreter while it is running the user program (here's the paper we wrote about this approach).
So it's just a tracing JIT but applied to the interpreter. One way to put it - it's effectively the same benefit as just running jython to begin with.
If you read the paper[1] linked in your quote, you'd see that it is not "just a tracing JIT"; the interpreter being run under the JIT has some special hooks that let it tell the JIT e.g. where the program counter is:
> Since the tracing JIT cannot know which parts of the language interpreter are the program counter, the author of the language interpreter needs to mark the relevant variables of the language interpreter with the help of a hint. The tracing interpreter will then effectively add the values of these variables to the position key. This means that the loop will only be considered to be closed if these variables that are making up the program counter at the language interpreter level are the same a second time. Loops found in this way are, by definition, user loops.
This is vastly distinct from how Jython works.
[1]: https://foss.heptapod.net/pypy/extradoc/-/blob/branch/extrad...
you say this and then quote directly what the novelty is and so i ask you - does that piece warrant a whole new term?
> This is vastly distinct from how Jython works.
jython isn't doing anything - it's a python interpreter that runs on the jvm. my point was that that's the same thing: an interpreter for a language that itself is being jitted.
In my opinion: Yes, definitely! Without meta-tracing techniques, a JIT'd interpreter can only hope to be on par with a compiled interpreter, never significantly faster. It will never go beyond the limitations of an interpreter, because the JIT can't "see" the user code that the interpreter is running.
I suppose if the tracing JIT were very complex and very tailored to a single language that might not make sense, but my impression of PyPy is that the opposite is true, and PyPy can in fact run other languages than Python.
there's nothing automatic about it. the user above you quoted from their paper - the user (person writing the interpreter) must annotate their code to pass hints to the tracing jit.
(It is possible more annotations are needed in PyPy than Graal, but still, an annotated interpreter is far, far simpler than writing an optimizing JIT!)
https://en.wikipedia.org/wiki/Partial_evaluation
Badass future space tech.
This is also a good illustration of how we ended up with the NULL problem. I don't think it's as big of a deal in this case as interpreters/vms/compilers are designed to be fungible in ways that source code was not, but it's something worth thinking on.
If he says yes to changing his name you have my full support.
And the big tradeoff is that the general JIT may be less capable of doing language-specific optimizations (indeed such optimizations have a chance to introduce bugs as the linked V8 blog shows, but they also can be correct and significantly improve perf in cases where the general JIT doesn't have the necessary info to do it itself).
Also the Graal team do some pretty advanced stuff to find security problems in the optimizations, in particular:
1. Lots of fuzzing.
2. Comparisons between Graal's output and the output of C2, which is a totally independent codebase. So you've got two different compilers and if they compute very different machine code, and it's not known to be an expected difference, that is used as a trigger to investigate things.
There are also small amounts of unsafe code in Truffle where checks are bypassed for speed, because it can be proven to be safe.
So overall it's a big win even if it can't eliminate 100% of all safety problems. This is similar to how Rust works, where unsafe stuff is done but in clearly defined sections that are easy to locate and audit, and the bulk of the unsafe code that most apps need is in the standard library.
BTW you can implement language specific optimizations in Truffle. It couldn't be competitive with V8 if that wasn't possible. For example dynamically typed scripting languages often need an optimization called object shapes. That's a part of the Truffle framework so all scripting langs can benefit from it. It's irrelevant for Java-like langs though.
Disclosure: I wrote the article.
*exception, the GC
Of course this all depends on your specific code and uses cases.
TruffleRuby even had the extremely big brain idea of running a Truffle interpreter for C for native extensions instead of actually compiling them to native code, just so that Truffle can optimize transparently across FFI boundaries. https://chrisseaton.com/truffleruby/cext/
TruffleC was a research project and the first attempt of running C code on Truffle that I'm aware of. It directly interpreted C source code and while that works for small self-contained programs, you quickly run into a lot of problems as soon as you want to run larger real world programs. You need everything including the C library available as pure C code and you have to deal with the fact that a lot of C code uses some UB/IB. In addition, your C parser has to fully adhere to the C standard and once you want to support C++ too because a lot of code is written in C++, you have to re-start from scratch. I don't know if TruffleC was ever released as open source.
The next / current attempt is Sulong which uses LLVM to compile C/C++/Rust/… to LLVM IR ("bitcode") and then directly interprets that bitcode. It's a lot better, because you don't have to write your own complete C/C++/… parser/compiler, but bitcode still has various limitations. Essentially as soon as the program uses handwritten assembler code somewhere, or if it does some low level things like setjmp/longjmp, things get hairy pretty quickly. Bitcode itself is also platform dependent (think of constants/macros/… that get expanded during compilation), you still need all code / libraries in bitcode, every language uses a just so slightly different set of IR nodes and requires a different runtime library so you have to explicitly support them, and even then you can't make it fully memory safe because typical programs will just break. In addition, the optimization level you choose when compiling the source program can result in very different bitcode with very different IR nodes, some of which were not supported for a long time (e.g., everything related to vectorization). Sulong can load libraries and expose them via the Truffle FFI, and it can be used for C extensions in GraalPython and TruffleRuby AFAIK. It's open source [1] and part of GraalVM, so you can play around with it.
Another research project was then to directly interpret AMD64 machine code and emulate a Linux userspace environment, because that would solve all the problems with inline assembly and language compatibility. Although that works, it has an entirely different set of problems: Graal/Truffle is simply not made for this type of code and as a result the performance is significantly worse than Sulong. You also end up re-implementing the Linux syscall interface in your interpreter, you have to deal with all the low level memory features that are available on Linux like mmap/mprotect/... and they have to behave exactly as on a real Linux system, and you can't easily export subroutines via Truffle FFI in a way that they also work with foreign language objects. It does work with various guest languages like C/C++/Rust/Go/… without modifying the interpreter, as long as the program is available as native Linux/AMD64 executable and doesn't use any of the unimplemented features. This project is also available as open source [2], but its focus somewhat shifted to using the interpreter for execution trace based program analysis.
Things that aren't supported by any of these projects AFAIK are full support for multithreading and multiprocessing, full support for IPC, and so on. Sulong partially solves it by calling into the native C library loaded in the VM for subroutines that aren't available as bitcode and aborting on certain unsupported calls like fork/clone, but then you obviously lose the advantage of having everything in the interpreter.
The conclusion is, whatever you try to interpret C/C++/… code, get ready for a world of pain and incompatibilities if you intend to run real world programs.
That's even true of the standard HotSpot-backed JVM. I've rewritten programs from Java to JS which has resulted in making them faster—because they're short-lived, and the JVM's slow startup chewing through the budget is never offset by any of the theoretical speedups that the JVM otherwise promises.
Probably having to do with the JVM being optimized for long-running server processes.
>the same results can be amplified if you use the JVM instead of AOT (i.e. it's even slower to start, but eventually it can be much faster.)
OpenJ9 has a caching JIT server that (theoretically) would work around this.
• GraalJS is today based on an AST interpreter, which are less efficient in general than bytecode interpreters. V8 starts by compiling JavaScript to an internal bytecode, interpreting that, then JIT compiling the hot spots. It's the same architecture as the JVM except that the bytecode isn't considered to be a stable or documented format.
• The Truffle JIT compiler is slower than V8's because partial evaluation adds overhead. Slower compiler = more time spent in the interpreter = slower warmup.
• V8 is heavily optimized for great startup time because web pages typically don't live very long.
The first problem is being tackled by adding a bytecode interpreter infrastructure to Truffle itself. In other words, the Truffle library will invent a bytecode format for your language, write the bytecode interpreter for it, then partially evaluate that to create the JIT compiler! It can also handle stuff like persisting the bytecode to disk, like .pyc files or .class files do. Moving all the stuff needed to implement fast languages into the framework is very much the Truffle way.
The second problem is harder to solve. There is supposedly a thing called (I think) the second Futamura projection, where you partially evaluate the partial evaluator, but IIRC that's very hard to actually implement and for server-side use cases, less important.
For anyone else who got curious: https://www.graalvm.org/latest/reference-manual/java-on-truf...
I’d only believe their perf claims if they used a more modern benchmark suite.
Oracle's profit motive is (AFAIK) enabling polyglot scripting inside their database. But Google has a profit motive of not having to pay Firefox or Safari a bigger chunk of the search engine ad profits.
So yes, I agree that the outdated benchmarks are a bit fishy. But if there was enough of a market I have no doubt that the Truffle/Graal devs would know how to put more funding to good use. There are alternative JVM runtimes that optimize for latency or start-up times. The Oracle JVM is optimized for their core market (long running server processes).
It's competitive in some respects. It's not used in browsers so it's probably not competitive there where there are different code patterns to server-side JS (unless you want a security upgrade more than top performance), due indeed to the slower warmup time which really hurts in the web environment where code gets discarded regularly.
Also, V8 probably has better memory usage. I haven't checked.
The warmup time is being worked on but these are research problems, and the V8 team is much larger than the GraalJS team is.
The memory safety claim is about as fishy as the benchmarks.
I'd trust the JIT hardening of a modern JS engine over whatever hardening has happened in the OpenJDK any day of the week. In other words: if the JS engine was built on top of Truffle, then the exploit technique would be all about finding bugs in Truffle/Graal/OpenJDK. And I bet that's easier than finding bugs in V8 or any other JS engine, just because the JS engines have been fighting the security fire with extreme prejudice due to their security exposure and I doubt that the OpenJDK crew has had to since it's not as worth it to attack them. And if it was as worth it to attack them, then the complexity of the OpenJDK would make it an easier target to attack and a harder target to meaningfully lock down.
- (Under W^X) User-land W->X transmutation support from the, i.e., pkey_mprotect(addr, len, PROT_EXEC | PROT_READ, pkey) against an * aligned_alloc()* region on Linux
- (Slower) Produce assembly and compile, or write an executable or library, and dlopen() it in an existing process or run it in a separate process and address space
- (Faster, least secure, not recommended) Write and execute against RWX pages directly