The Challenge of Cross-Language Interoperability (2013) | Hacker News Reader

The Challenge of Cross-Language Interoperability (2013)

queue.acm.org

45 pointsmpweiher23 comments

“We are sorry ...

... but we have temporarily restricted your access to the Digital Library. Your activity appears to be coming from some type of automated process. To ensure the availability of the Digital Library we can not allow these types of requests to continue. The restriction will be removed automatically once this activity stops.

We apologize for this inconvenience.”

Anyone else getting this?

If so, have a Wayback Machine link: http://web.archive.org/web/20180219210654/https://queue.acm....

I got that too, despite being already logged in to the site. I clicked on the PDF logo in the upper right corner, though, and was served the paper. Some bug, I guess.

I see it too.

I've been seeing it since yesterday and that "C is not low level" post. I was at a different ip then too.

A lot of this has to do with languages in no way designed for interoperability. There have been platforms designed for cross-language development.

OpenVMS standardized calling conventions for interoperability. CLR does that at a VM level in low-level way. Ten15 on FLEX machine did it in high-level way. Lots of variants of languages are targeting C, JVM, etc structured in ways that make interoperability easier. Julia making using C and Python code easy is an example. Finally, there's metaprogramming tools that make all the languages DSL's interoperable via core language. Racket, Alan Kay's stuff, and sklogic's tool cone to mind.

The ones having trouble are usually evolved away from cross-language development with a lot of complexity that makes it harder than it already is. Of course they're having to resort to things like ZeroMQ or whatever.

I'm on the lookout for a language that does a) not itself require a VM (can be properly compiled) or significant runtime, but b) can be hosted in a VM without too many leaky abstractions, and c) can interface well with other ABIs and VMs.

C++ comes close (e.g. with Windows Runtime and Objective-C++) but its memory model fails (b) and there are no type generators or such so you get a lot of glue code generation.

A small Lisp dialect could easily be hosted in any VM and it can be made to interface well with just about anything transparently (look at Racket's C bindings) but the dialects I've seen fail (a) and lack lifecycle hooks that would make building a C++0-like com_ptr<> type possible.

There's no standardisation of how to represent ownership across ABI boundaries yet, and different languages' models are fundamentally incompatible: either the VM/runtime expects to own all memory management (in which case interfacing with another VM/runtime that follows the same model is virtually impossible) or the programmer does it manually (perhaps with some help from the language in the case of C++ or Rust). Embedding a language that does the latter in a VM language is always going to involve fiddling with memory ownership in a language unsuited to it (as you do when using SWIG or the like).

What are you actually trying to achieve here? If you're writing a library for use from both VM and non-VM languages, you'll want your library functions to have clear/simple memory ownership semantics anyway, and at that point any unmanaged language will work reasonably well for you. If you just want your library to be usable from several different styles of language, maybe a multi-language VM will work for you.

I think templated or generic handle classes are pretty good for dealing with foreign objects. They can handle any bookkeeping a VM requires.

The background is that I've been playing with C to write Win32 apps that are highly backwards compatible (back to 3.11/Win32s if possible) but also modern when running on newer Windows versions.

For the latter I've been manually writing COM exports which was painful but it works. This made me consider C++ as an alternative where I'd be able to use templated types (and destructors!) to great effect. I can limit my use of the standard library to keep it freestanding.

However I would still be limited to writing native code, which is why I'd like something that can target the Android Runtime and such.

> I think templated or generic handle classes are pretty good for dealing with foreign objects. They can handle any bookkeeping a VM requires.

What do you mean by this?

With C++' destructor call guarantee and its templating features, it's feasible to implement shared_ptr-like types for foreign object types. Microsoft's ComPtr for example takes care of COM referencing counting. If some VM would require you to pin and unpin objects before and after use, you can do that in such a handle.

Hmm, you're talking about calling into a VM language from a manual-ownership language like C++? At that point any VM language would be fine, but I'm not sure that scenario makes sense in the first place - if the outer program is in C++ that's presumably because it has some requirement avoid GC, and embedding a VM that has to do GC would undermine that.

You are badly conflating some issues here. How to implement automatic memory management is a runtime design issue. How to enforce proper non-memory resource management is a language design issue. Nothing forbids an implementation of a safe-Rust-like language with a garbage collector. Destructors would still be called deterministically, and destructed objects would still be unusable afterwards, as mandated by the language's semantics. But memory will only be reclaimed during the next garbage collection cycle.

There are other (better!) reasons against cross-language interoperability, though, such as the reduction in static guarantees to an unusable lowest common denominator.

You seem to be asking for contradictory things. For instance, low runtime overhead implies manual memory management but running within a VM implies automatic memory management. What about a "bundled" runtime like Go, some Lisps, or Ocaml? Lua?

Your c) needs refinement. It's not just memory ownership but also idiomatic use of the language. Heavy on objects? Functions? Mutable or immutable (don't forget strings)?

I think the memory management issue can be addressed by using reference counting semantics in the language (wrt aliasing, scoping and weak vs strong references). That maps well to both manual management and garbage collection.

Some more searching the web lead me to Haxe. I'll have a look at how they handle these issues.

I originally wrote a long reply, but my damn tablet crapped out on me and I lost it, so my apologies for the brevity here.

Object Pascal will do most of what you want ( c) with VM interop is an issue, but will be for most languages), and is available on a lot of platforms.

On native platforms, you get straight-up manual memory management of class instances (Create/Free methods) or alloc/free memory directly.

On VM platforms like the JVM or CLR, or LLVM-backend compilers that use ARC, the Create/Free methods for class instances are typically mapped to Create/Dispose methods so that non-GCed resources can be disposed of properly.

Things like strings, dynamic arrays, and interfaces are already reference-counted and don't use manual memory management, but can be move'd, etc. and dealt with like normal blocks of memory on native platforms.

There is also marshalling for lower-level access to C APIs from the VM platforms and LLVM back-ends, and direct call access to C APIs from native code with various call modifiers.

There are also some neat things for native platforms like methods for hooking class instance allocation so that you can provide your own per-class memory pools.

The only downside is that you may find some syntax differences on certain platforms, and besides Free Pascal, the compilers tend to cost $$$.

This is a very interesting idea and might be just what I'm looking for. Thanks.

The Java ecosystem's JSR 223 is something to look at (at least for hosted languages with a canonical object system).

not as impressive as Graal.

Yes, sort of. Or maybe kind of not?

I am not sure I'd call what Graal is doing "interoperability", because you essentially pull everything into that VM, instead of interoperating with it where it is.

So they "interoperate" with C by making a C compiler that creates a Graal-compatible AST, and as far as I can tell, the only way of interoperating at this point is creating a new Graal compiler.

What I've gathered about Graal is that once you have a compiler for the platform, Cross-Language communication on a hot path gets optimised to near-zero cost.

Exactly. You're not really crossing boundaries any more, you just have different surface-syntaxes for "the Graal language system".

well your statement is neither true nor is it false.

See: http://www.graalvm.org/docs/reference-manual/languages/llvm/ for example. Basically you generate bitcode and can run the bitcode on graal. However you can still load native libraries into Graal and access them, you can't probably access them directly in java without a wrapper (I'm not sure about that yet).

So basically you compile your C/C++/Rust Code to BitCode instead of Object Files/Binaries and run that BitCode on Graal.

I.e. with java:

Context polyglot = Context.newBuilder() .allowAllAccess(true) .option("llvm.libraries", "/usr/lib/libcurl.dylib") .build();

        Source source = Source.newBuilder("llvm", new File("hello.bc")).build();
        Value cpart = polyglot.eval(source);
        cpart.getMember("main").execute();

Whereas hello.gc is: #include <stdio.h> #include <curl/curl.h>

int main() { CURL *curl = curl_easy_init(); if(curl) { CURLcode res; curl_easy_setopt(curl, CURLOPT_URL, "http://example.com"); res = curl_easy_perform(curl); curl_easy_cleanup(curl); }

    printf("Hello from GraalVM!\n");
    return 0;
}

I have used one, but not the other, so this is not a comment on whether Graal is as good or not as good, but along the line "they did it right with the previous system, so there's hope to get it right this time, too".

Reply on news.ycombinator.com