Hacker News Reader

Comment by pjmlp | Hacker News Reader

You do it the standard way, package them inside the jar file.

Oh, does this actually work?

I was on the assumption that it was dynamically linking the libarary with the OS dynamic linker, which in no OS I'm aware of is capable of loading libraries inside of zip files.

Not sure where I got that notion. Maybe I was overthinking this.

Yes. Check out a library like zstd-jni. You'll find native libraries inside it. It'll load from the classpath first, and then ask the OS linker to find it.

I'd like to learn how they do it. Because last time I've looked at this, the suggested solution was to copy the binaries from claspath (eg: the jar) into a temporary folder then load it from there. It feels icky :)

Yep, you're right, they do exactly that. Apologies for the confusion.

Decompiled class file:

    try {
        var4 = File.createTempFile("libzstd-jni-1.5.0-4", "." + libExtension(), var0);
        var4.deleteOnExit();

This wouldn't work on Windows, because you can't delete a DLL while it's in use

You might be able to use FILE_FLAG_DELETE_ON_CLOSE, but this would likely require calling the Windows API functions directly.

Couldn't you: Extract DLL Load DLL Unload DLL Delete DLL ?

Though in the example given, I do see your point now. You'd have to make sure the DLL was unloaded before the delete-on-exit happened.

According to JNA it's not safe to unload the DLL:

https://github.com/java-native-access/jna/blob/40f0a1249b5ad...

  Do NOT force the class loader to unload the native library, since
  that introduces issues with cleaning up any extant JNA bits
  (e.g. Memory) which may still need use of the library before shutdown.

Following the blame back to 2011, they did unload DLLs before https://github.com/java-native-access/jna/commit/71de662675b...

  Remove any automatically unpacked native library.  Forcing the class
  loader to unload it first is only required on Windows, since the
  temporary native library is still "in use" and can't be deleted until
  the native library is removed from its class loader.  Any deferred
  execution we might install at this point would prevent the Native
  class and its class loader from being GC'd, so we instead force 
  the native library unload just a little bit prematurely.

Users reported occasional access violation errors during shutdown.

You can install a shutdown hook to do cleanup like this.

    Runtime.getRuntime().addShutdownHook(...)

That's how java.io.File#deleteOnExit works under the hood. The DLL is still loaded at that point and can't be deleted.

Ah, looking through the docs [1]; you have to use your own ClassLoader (so it can be garbage-collected), and statically-link with a JNI library which is unloaded when the ClassLoader is garbage-collected.

1: https://docs.oracle.com/en/java/javase/22/docs/specs/jni/inv...

Hmm, interesting. They do have DLLs in the JAR...

EDIT: Disregard. I am wrong. Original below.

You can just load as a resource. We do this internally since much of network stack is C. But we use JNI because code is older than Java 22.

You made me search it again. And still I don't see how that's possible. `Runtime.load` requires a regular file with an absolute path[0].

Stackoverflow is full of "copy it into a temp file" solutions. ChatGPT keeps saying "sorry" but still insists on copying it into a temp file :)

[0] - https://docs.oracle.com/en%2Fjava%2Fjavase%2F22%2Fdocs%2Fapi...

Embarrassing of me to give you wrong answer. I went and checked my old code and:

     new FileOutputStream(tmpFile)

Apologies.

Sounds promising.

I have some extremely unwieldy off-heap operations currently implemented in Java (like quicksort for 128 bit records) that would be very nice to offload as FFI calls to the corresponding a single-line C++ function.

Why not give C# a try instead? It has everything you ask for and then some.

Because "some inconvenience/unmet requirement" from a language is not an invitation to "throw out the whole platform and your existing code and tooling, and learn/adopt/use an entirely different, single-vendor platform".

Except if we're talking about some college student or hobbyist picking their first language and exploring the language space...

He would still have to call out to the C++ function.

Assuming it is "sort for 128bit records", that's something C# does really well - writing optimized code with structs / Vector128<T> / pointer arithmetic when really needed without going through FFI and having to maintain separate build step and project parts for a different platform.

But even if it was needed, such records can be commonly represented by the same structs both at C#'s and C++'s sides without overhead.

An array of such could be passed as is as a pointer, or vice versa - a buffer of struts allocated in C/C++ can be wrapped in a Span<Record128> and transparently interact with the rest of standard library without having to touch unsafe (aside from eventually freeing it, should that be necessary).

Wow, you all are sure mad enough to go out of your way and downvote my comments elsewhere.

Stay in the swamp :)

I remember using Sqlite Java and not having to install sqlite on the image. Then I looked inside the Sqlite-java's jar and they just packed the sqlite binaries for the different OSs in the jar!!

Not sure if this is still the case, but one of the Java Sqlite driver used something called NestedVM to run Sqlite in the JVM when a native library wasn't available. It worked by cross-compiling the code to mips, then transpiling the mips assembly to Java byte code. I can't remember if it bridged system calls or libc calls to Java for things like file IO.

I once (2016 ish) used a serial-port library for Java. Needed to be cross platform desktop app for Linux, Windows and Mac (in that order, all on x86/64). And it was. I have forgotten the name of the library project I included, but it included DLL binaries for the platforms we were targeting.

That's a common solution. I do the same.

Android knows how to do this, actually.

Is there a solution when the binaries are 500mb+ per platform?

People seem pretty happy when Go and Rust do the same with static linking, advocating how great it happens to be.

You can't be as aggressive at removing functions in Java than in Rust though since it's dynamic dispatch (e.g., if you use toString once in your code, you need to keep all implementations of toString which are reachable even if users don't use reflection).

.NET's trimmer/linker deals with this quite well, only referenced or otherwise observable .ToString() implementations are rooted.

Without it 1.6-2MiB-sized AOT binaries would not have been possible (most space is occupied by standard library/runtime bits and GC)

Except that is what jlinker, and GraalVM/OpenJ9 (among other AOT toolchains) do in practice.

In Java libraries are shared precompiled so the package manager either needs to be platform aware or distribute fat bundles.

Only for those that are yet to learn how to use jlinker.

Static linking in Go and Rust includes compiled code only for the target platform. It does not include compiled code for every possible architecture, including 32-bit MacOS and Solaris on PowerPC.

Solaris... now thats a name I have not heard in a long time. A long time.

Hence why you end up with 300MB x platforms, and tricks like upx.

Libraries like JCEF have support tools to download the libraries either at runtime or during the build, to offload from Maven Central.

Reply on news.ycombinator.com