To start with, it eliminates the IPC and Sidecar overhead of using something like Ollama. It might seem trivial when comparing the speed of a REST request to that of inferencing, but the ms add up at scale. And for what I'm doing higher up the stack, every nanosecond I save here, gives me headroom to do cooler things. It also allows you to make lots of smaller round trips to your AI generating layer. Even if you're just operating REST on the loopback layer, there is overhead and kernel-level context switching that adds up. I can go further into this if you'd like!

Once you make the decision to forgo IPC and do things in house, you typically would go the JNI route. Project Panama in JDK 22+ however opens us up to direct memory access at the lowest of levels to shave even more cycles. We get direct, off-heap memory access inside the same process space. This allows us to do some cool things like creating a memory and GPU governor with more direct access to the hardware its controlling, while completely bypassing the Java GC and its overhead! It also allows us to do direct memory copying without ever going through java heap allocations nor primitive array copying.

With this code, I mapped the native C-structs directly to Java objects to bridge the ABI gap cleanly. The naive way to do so, without writing the argus c layer, and interfacing directly with llama.cpp. By having a static, immutable ABI, you save yourself the headache of having to rewrite your entire brittle Panama bindings every time upstream changes a single variable. Then I went a step further and provided the Java API for you directly to interface with!

Idk about you, but if you've ever had to write a JNI wrapper, I much prefer an easy to use clean Java API ready-made.

Then I package everything inside one nice friendly JAR, allowing jvm developers to have access to multi-modal inference with one import.