Comment by lvh | Hacker News Reader

Comment by lvh | Hacker News Reader

The JVM has a number of cool features that enable you to efficiently drop down into C (FFI) yourself: these tricks aren't just for built-ins.

A quick overview:

- First there was JNI. JNI means that you write method stubs with the "native" keyword. Then you run javah, which gives you some C glue code that you eventually need to compile. This is very fast, but it's annoying because now you need a tool chain everywhere. The Python equivalent of this is roughly writing CPython extensions.

- People thought JNI was annoying, so Sun developed JNA. JNA lets you bind a library directly: all the magic comes with the JVM, and you can just dlopen something and call some syms. This works fine, but it's very slow. The Python equivalent of this is roughly ctypes.

- Most recently, there's JNR and jnr-ffi. They do a very clever trick: you use JNI to get to libffi, and then you use libffi to call everything else performantly. You get roughly the performance of JNI, with the convenience of JNR. The Python equivalent of this is roughly cffi.

JNR is way more usable than I thought it would be. I develop caesium[0], a Clojure libsodium binding, using jnr-ffi and a pile of macros. I gave a talk about this at Clojure Conj (recording [1], slides [2]) if you're interested.

To be fair: this code uses intrinsics, which means that it's implemented differently than the three methods shown above, so it's still slightly different. It's just not different in a way that's meaningful to you unless you're working on the JVM itself :)

[0]: https://github.com/lvh/caesium [1]: https://dev-videos.com/videos/Lf-M1ZH6KME/Using-Clojure-with... [2]: https://www.lvh.io/CCryptoClojure/#/sec-title-slide

If folks are interested in this, I figured some folks might like our c++ version of this (been around for 5 years ish now): https://github.com/bytedeco/javacpp

Also comes with tons of pre cooked projects already ready to go: https://github.com/bytedeco/javacpp-presets

We use it in production at skymind and it powers our whole cpu and gpu stack with built in memory management among other things.

Also has maven and sbt plugins so you can plug it right in to your workflow automatically just maintaining the java code.

One of the coolest features is the name matching so you can get semi automatic mapping.

We auto generate our JNI bindings from this.

JNR (and JNA and cffi) strike me as unsafe due to the lack of type enforcement. Systems like this let you call any random pointer as if it were a C function of any type. That usually works, but sometimes doesn't. If you're lucky, mistakes make your program blow up right away. If you're unlucky, you get impossible to diagnose memory corruption.

I'd much rather write conventional bridges and have the system check that I'm right.

To be fair, when you're writing the C side of your JNI bindings, it'll also let you call any random pointer as if it were a C function of any type. You'll have non-type-safe code _somewhere_, it's just a choice of where.

Ah, not true! You'll have as much type-checking as C allows for, which is actually quite substantial. Moreso in C++. The opportunities for accidental type mismatch are much reduced: basically, to the JNI->function bindings (eliminated if you codegen the glue).

I see what you’re saying, but that piece of code doesn’t exist at all in JNR (so it can’t have bugs) and you can codegen JNR/FFI too to eliminate the same hole, right?

I feel like saying “don’t mess up this signature” is not only plausible but it’s a lot better than having to deal with writing and compiling the stub for every platform.

This is why pycparser exists, for example. https://github.com/eliben/pycparser

The solution is to have some tool which parses C header files and uses them to build the Java bindings.

Unfortunately, C is a pretty nasty language to parse, so you end up using something like http://www.swig.org/ or https://github.com/rpav/c2ffi to parse it for you. But the challenge with adopting those sort of tools for Java is they aren't written in Java, they are written in C and/or C++. (Obviously that doesn't stop you from using them with Java, but it does make the whole thing less pleasant.)

There is a project to get something like JNR bundled with the JDK – see http://openjdk.java.net/jeps/191 and http://openjdk.java.net/projects/panama/

> Then you run javah, which gives you some C glue code that you eventually need to compile.

That's deprecated and no longer supported in Java 10. The recommend way is to use javac -h.

> This is very fast

There is a certain call overhead for JNI that JNA and JNR necessarily share as well. As a normal Java application you don't really have a way to get around this.

See https://stackoverflow.com/questions/36298111/is-it-possible-...

> but it's annoying because now you need a tool chain everywhere.

You only need a toolchain for building, you can just ship the .so. These days you get pretty far with Linux 64bit, macOS 64bit and Windows 64bit.

Reply on news.ycombinator.com