mukel

This is a port of NVIDIA's Parakeet model family that runs end-to-end on the JVM. And it's fast, competitive with the native engines on CPUs. Steve Jobs' 15-minutes Stanford speech transcribes in ~15s in my laptop, ~5s on my workstation, using the 110M Q4_K model.

I built llama3.java in the past, this is a follow-up: Gemma 4 running entirely on the JVM.

No Python. No JNI. No native code. Just Java.

It’s (mostly) a single Java file implementing the full stack:

GGUF parsing, tokenization, Gemma 4 transformer inference, quantizations, CLI...

Built using the Java Vector API, with support for GraalVM Native Image.

Code: https://github.com/mukel/llama3.java

Features: - Single file, no dependencies - GGUF format parser - Llama 3 tokenizer - Support Llama 3, 3.1 (ad-hoc RoPE scaling) and 3.2 (tie word embeddings) - Fast matrix-vector multiplication routines for Q4_0 and Q8_0 quantized tensors using Java's Vector API - GraalVM's Native Image support - AOT model preloading for instant time-to-first-token

Llama3.java: featuring .GGUF file format support, Q8_0 and Q4_0 quantizations, fast matrix/vector multiplication routines using Java's Vector API; served by a simple CLI with a --chat mode to interact with the Llama 3 models.

This will boost adoption at so many levels: - Importing a Truffle language as a regular Maven dependency - Ease integration with mainstream package managers - Ability to update to the latest language version, independently of the JVM used

I'm still in awe at how smooth the Truffle "unchaining" worked out with no API changes (just a few necessary additions).

Author here: I implemented several versions of matmul with different unrolling schemes using the Vector API and I got a ~4X speedup with a single thread, but the speedup fades the more threads you add. I think that performance is constrained by memory bandwidth which is saturated with a small number of threads, regardless of vectorization.

A Java port of llama2.c that performs very close to C on large models. Llama 2 7B runs at a whooping 1.6 tokens/s.

GraalVM team member here. Implementing any mainstream language is indeed a challenge, more so if you have to maintain bug-compatibility and cope with all the bits of bad design that went through the cracks in the de-facto implementation. Truffle is not for beginners, but knowing the basic set of features e.g. partial evaluation, deoptimization... can get you very far already e.g. you can easily speedup any interpreter by 10X or more with minimal changes.

How long does take to implement a programming language? Well, from hours to years... depending on the language. To make my point; how long would it take to implement a JVM? A JVM is a complex beast, so I would myself guess from years to a decade probably, what if I told you, that Espresso was written in just 6 months by an intern and a seasoned engineer... in just 6 months it was able to Minecraft and even run itself. I assure you there's no magic here, and certainly no blinding talent either; the only reason for this unheard productivity was Graal/Truffle. So, whenever I talk about Espresso I always give all credit to Graal/Truffle, it is a sublime platform for implementing fast languages and runtimes, of which Espresso is just a byproduct.

The goal is not to compete, but rather complement HotSpot/GraalVM with:

- (Polyglot) scripting with Java

- Augmenting native images e.g. native javac with instant startup + annotation processors (very dynamic) running on Espresso

- A simple non-invasive JVM for constrained environments

- DCEVM-like features for developers

- Approachable academic playground

- Fast prototyping of JVM features e.g. it took our intern just two weeks to implement invokedynamic/MethodHandles

It's a Linux-only trick: https://github.com/kt97679/tetris/blob/52dfb3a703e4dd5b37990...

Java on Truffle is a normal Java application, it can run on a vanilla OpenJDK, doesn't need the bootstrap step.

Currently Graal/Truffle implements the first Futamura projection: you give it an interpreter written in Java, it automatically generates a compiler (JIT). In this mode we have to keep the interpreter IR graphs (blueprints) around for partial evaluation.

With the first Futamura projection: Partial evaluator + interpreter + user code.

There's active work on implementing the second Futamura projection, where you partial evaluate the partial evaluator with respect to the interpreter, generating a specialized partial evaluator for that interpreter. With the second Futamura projection: (Specialized partial evaluator + interpreter) + user code.

This is truly fascinating and beautiful and the fact that it works for Java and not just a toy academic prototype is mind-blowing.

Running on HotSpot is an advantage for the development of Java on Truffle itself; the tooling is amazing, debugger, inspectability, you can debug the VM as a normal Java application... GraalVM can compile an AOT version of Java on Truffle, then no JVM is needed.

Java on Truffle is an implementation of the Java Virtual Machine Specification, Java SE 8 and Java SE 11, built upon GraalVM as a Truffle interpreter. It is a minified Java VM that includes all core components of a VM, implements the same API as the Java Runtime Environment library (libjvm.so), and reuses all JARs and native libraries from GraalVM. The project name behind this implementation is "Espresso". Its open source version is available on GitHub.

Google is going downhill, the day I finished my internship I sweared I'll never in my life work on any money-making, no-challenge project. A bunch of engineers do enjoy what they do, they work on the cool projects, that's enough to keep it going; the rest is just cattle, work for the cash, enjoy the free food and the reputation of working at Google; I'm still sick of being bombarded with the "changing the world" nonsense. Change projects/company, find a mentor and/or a mentee, build new stuff; find your purpose, unleash your intellect. Don't fall for the "changing the world" lie.