KingJoker

Glad to hear! Let me know if you run into any issues!

You're welcome, I appreciate the engagement!

Your intuitions are right, faster switching between agent calls, and tighter packing of agents in the same amount of compute space.

There's a second thing besides just efficiency that I provide here however. At a developer-user level, it automagically provides JVM developers (not just java) a frictionless way to add AI into their stack with a single dependency.

It makes using AI in java as easy as it is in python.

I'm providing the [what I've found to be missing] layer for AI in java in the most efficient way I could.

I'm not sure if by user you meant developer (user of this code) or user as in end-user. Let me know if I still haven't answered your question fully yet!

I addressed this a little bit in the comment below, but the cycles add up. I'm doing some pretty crazy things higher up in the stack, that I'm not quite ready to release yet. But even micro optimizations here add up at the scale I'm working at to allow me the headroom I need. I'm relying on a high-frequency recursive agentic loop that chokes a real-time guarantee without every optimization I can give it.

I started by removing the IPC overhead from a weaviate db connection, and doing all my vector math in house with a lightweight sqlite db. This became the next obvious target for optimization once I saw how much that saved me doing things in-house.

To start with, it eliminates the IPC and Sidecar overhead of using something like Ollama. It might seem trivial when comparing the speed of a REST request to that of inferencing, but the ms add up at scale. And for what I'm doing higher up the stack, every nanosecond I save here, gives me headroom to do cooler things. It also allows you to make lots of smaller round trips to your AI generating layer. Even if you're just operating REST on the loopback layer, there is overhead and kernel-level context switching that adds up. I can go further into this if you'd like!

Once you make the decision to forgo IPC and do things in house, you typically would go the JNI route. Project Panama in JDK 22+ however opens us up to direct memory access at the lowest of levels to shave even more cycles. We get direct, off-heap memory access inside the same process space. This allows us to do some cool things like creating a memory and GPU governor with more direct access to the hardware its controlling, while completely bypassing the Java GC and its overhead! It also allows us to do direct memory copying without ever going through java heap allocations nor primitive array copying.

With this code, I mapped the native C-structs directly to Java objects to bridge the ABI gap cleanly. The naive way to do so, without writing the argus c layer, and interfacing directly with llama.cpp. By having a static, immutable ABI, you save yourself the headache of having to rewrite your entire brittle Panama bindings every time upstream changes a single variable. Then I went a step further and provided the Java API for you directly to interface with!

Idk about you, but if you've ever had to write a JNI wrapper, I much prefer an easy to use clean Java API ready-made.

Then I package everything inside one nice friendly JAR, allowing jvm developers to have access to multi-modal inference with one import.

Thanks! It was a fun challenge getting an AI agent to pair with me while adhering to strict c-struct padding and memory alignment.

Let me know if anything sticks out you'd like to discuss deeper!