Lessons from debugging a tricky direct memory leak

I’m incredibly confused why profiling wasn’t even mentioned in this post.

I would highly recommend anyone troubleshooting something like this to do continuous profiling and the root cause of this type of problem becomes painfully obvious within seconds.

(Disclaimer I founded a company in this space, and we recently open sources a library for rust to do this: https://www.polarsignals.com/blog/posts/2023/12/20/rust-memo...)

We ran into a problem where we blew the direct memory limit recently.

To trace the problem, I added a breakpoint on ByteBuffer.allocateDirect() - this is how DM is normally allocated. But rather than suspending the process, we set it to print the stacktrace and continue.

This gave us handy output of possible causes. Focusing on the cases which were called from our own code (likely to be the root cause), we found a call to Files.readAllBytes() - which immediately sounds suspicious to cause a memory leak. Indeed this was the culprit, and explained why the problem was sporadic and hard to reproduce (there is some kind of per-thread caching behaviour of direct memory I never did understand properly).

Incidentally, Files.readAllBytes() uses direct memory because it uses an area of memory which both the OS can access (onto which to read the file contents) and which also Java can access (for the application code, rather than copying to the heap, which could be inefficient for large files).

Fortunately, this "by inspection" approach got us to the answer.

Others have stated profiling would work well. We'd have needed a local reproduction of the issue, which wasn't available at the time (could not reproduce without realistic workloads). Otherwise we could have used our SaaS APM software which runs in deployed envs, but that takes a sampling approach and doesn't really dig deep enough for this kind of stuff.

If you have a memory leak, then take a process that has been leaking for a while and look at its memory. Just pole a few places at random and statistically you should quickly find what is leaking. I am pressure this would have cut down a lot of the investigation needed in this story (though it is always easy to say this kind of thing after the biug has been found)

I stumbled upon this article in the latest edition of the 'Java Performance Tuning Newsletter'. Relatively recently, I also published a blog article [1] diving into all aspects of JVM memory, including native memory profiling.

[1]: https://serce.me/posts/01-02-2023-jvm-field-guide-memory.

Off topic, but I use Pinterest extensively for work and pleasure. It has all sorts of funny bugs that don't really interfere with using it. One of them is content that looks spicy, but is grandfathered in so it doesn't get flagged. If you try to repin it on your board their AI gets upset. These two tricky images caught it out this week for me:

https://imgur.com/a/9Pew0EE

Java flight recorder wouldn't help here?