You can use native memory tracking to better inspect offheap usages. I had a similar problem where the default MaxMetaspaceSize was excessive and adjusted it to a reasonable setting.

-XX:NativeMemoryTracking=summary

jcmd <pid> VM.native_memory summary