Show HN: Low-latency local LLM runner via OpenJDK Panama FFM (Java 22) | Hacker News Reader

Show HN: Low-latency local LLM runner via OpenJDK Panama FFM (Java 22)

github.com

38 pointsKingJoker10 comments

I wanted to run AI from inside the JVM. I started out with the standard REST sidecar, ripped that out to use Project Panama (Foreign Function & Memory API) in the new JDK versions to interface directly with llama.cpp. I still wasn't happy with how that functioned, so I built libargus.cc to get a clean ABI to expose a structured API up in the JVM landscape. It still uses Project Panama to interface directly with llama.cpp, whisper.cpp, and ggml compute graphs.

I have zero-allocation on the hot paths, memory segments for prompts and tokens are allocated once inside confined Arenas. Raw pointers pass straight through down to the low C level. This avoids primitive array cloning and heap churn.

I mapped out the native structures from llama.cpp and whisper.cpp while matching the compiler's padding to maintain safe memory access.

I bundle pre-compiled native binaries in the jar for easy deployment.

This execution engine provides the foundation I need for work I'm doing on a spatio-temporal memory layer (L-TABB) to replace RAGs. I'd love to get technical feedback to polish any issues while I continue working on the next layer. Deep-dives from anyone hacking on Project Panama or low-latency systems in modern JDK would be very appreciated!

I'm much better with code than prose, so I'll let the code do most of the talking.

Happy Hacking! /David

Code: https://libargus.cc Project Landing Page: https://projectargus.cc

Open articleView on HN

I'm curious to hear what bottlenecks you encountered in the traditional path. Of all the compute and data shuffling involved in LLM inference, I would have thought shuffling the raw input/output around would have been a trivial part of the overall cost, and thus not a big optimization target?

I addressed this a little bit in the comment below, but the cycles add up. I'm doing some pretty crazy things higher up in the stack, that I'm not quite ready to release yet. But even micro optimizations here add up at the scale I'm working at to allow me the headroom I need. I'm relying on a high-frequency recursive agentic loop that chokes a real-time guarantee without every optimization I can give it.

I started by removing the IPC overhead from a weaviate db connection, and doing all my vector math in house with a lightweight sqlite db. This became the next obvious target for optimization once I saw how much that saved me doing things in-house.

What are the benefits of this. I get the impression it improves speed of…something? Most of the time when using AI comes from the LLM, so I’m curious what this improves?

To start with, it eliminates the IPC and Sidecar overhead of using something like Ollama. It might seem trivial when comparing the speed of a REST request to that of inferencing, but the ms add up at scale. And for what I'm doing higher up the stack, every nanosecond I save here, gives me headroom to do cooler things. It also allows you to make lots of smaller round trips to your AI generating layer. Even if you're just operating REST on the loopback layer, there is overhead and kernel-level context switching that adds up. I can go further into this if you'd like!

Once you make the decision to forgo IPC and do things in house, you typically would go the JNI route. Project Panama in JDK 22+ however opens us up to direct memory access at the lowest of levels to shave even more cycles. We get direct, off-heap memory access inside the same process space. This allows us to do some cool things like creating a memory and GPU governor with more direct access to the hardware its controlling, while completely bypassing the Java GC and its overhead! It also allows us to do direct memory copying without ever going through java heap allocations nor primitive array copying.

With this code, I mapped the native C-structs directly to Java objects to bridge the ABI gap cleanly. The naive way to do so, without writing the argus c layer, and interfacing directly with llama.cpp. By having a static, immutable ABI, you save yourself the headache of having to rewrite your entire brittle Panama bindings every time upstream changes a single variable. Then I went a step further and provided the Java API for you directly to interface with!

Idk about you, but if you've ever had to write a JNI wrapper, I much prefer an easy to use clean Java API ready-made.

Then I package everything inside one nice friendly JAR, allowing jvm developers to have access to multi-modal inference with one import.

Thank you for the detailed and technical response. I’m not doubting it providers benefits at the technical level, and I’m keen to dive into it deeper, but I was hoping to understand what benefits this translates to at a user level. I’m imagining something like it can lead to faster switching between between agent calls, or even at a developer level, it allows you to run agents/llms in a more constrained environment, therefore you can run more agent containers in parallel?

I’m familiar with enough with Java that I can squint and follow some of it, but I’m hoping to understand more at a macro level before committing to diving into the technical details.

Again, thank you for the information already provided.

You're welcome, I appreciate the engagement!

Your intuitions are right, faster switching between agent calls, and tighter packing of agents in the same amount of compute space.

There's a second thing besides just efficiency that I provide here however. At a developer-user level, it automagically provides JVM developers (not just java) a frictionless way to add AI into their stack with a single dependency.

It makes using AI in java as easy as it is in python.

I'm providing the [what I've found to be missing] layer for AI in java in the most efficient way I could.

I'm not sure if by user you meant developer (user of this code) or user as in end-user. Let me know if I still haven't answered your question fully yet!

Great. I feel like this is something that I’m going to dig into more. At day job I work with a platform that’s built on OSGI and JCR. I’m keen to explore if this could enable any interesting interactions there (they might not necessarily be good interactions though haha).

Glad to hear! Let me know if you run into any issues!

This is pretty impressive.

Thanks! It was a fun challenge getting an AI agent to pair with me while adhering to strict c-struct padding and memory alignment.

Let me know if anything sticks out you'd like to discuss deeper!

Reply on news.ycombinator.com