Hacker News Reader

Related: There recently was a Twitter thread that you might be interested in: It's about the need for a new ABI which covers the middle ground of Rust, Swift, Zig, Nim, and other mid/low level languages, with lifetime annotations baked in so that the language can read them and manage memory without the need of developer's manual help.[0]

[0]: https://twitter.com/bitshiftmask/status/1286411475658178561

> Wouldn't it be better to start working on an "LLVM for language interoperability"? That way every language only needs to provide a connection to an intermediate specification, and it can connect to any language supporting that same specification.

My somewhat limited understanding is that the WASM interface types proposal is hoping to serve that sort of role. Note that in this case, the "web" part of "web assembly" is kind of a red herring; as I understand it, this proposal doesn't really have anything to do with the web and is more of an attempt to utilize WASM to create a sort of "universal ABI" that can replace the current de facto choice of using the C ABI everywhere.

it comes with some overhead though, which is useful in many circumstances as it also gives some sandboxing, but if your goal is universal interop then you probably want to have a solution that doesn’t incur a speed penalty.

We could call it something like "Common Language Infrastructure" or COM maybe.

And then remember that Objective-C is, essentially, COM with language support.

Unfortunately COM was never OS independent so that kind of killed it off. It was also runtime instead of compile-time.

Why the past tense? COM is alive and well, being the way the large majority of Windows APIs get introduced since Vista.

While COM is Windows specific, other implementations with similar ideas do exist.

IBM and Unisys mainframe language environment, gRPC, D-Bus, XPCOM, SOM, Android IPC, Fuschia IPC.

Indeed, COM has some right ideas. There's just too much influence from the mid-90's OOP hype in it.

What? COM is the most successful implementation of ECS, I thought it was fashionable nowadays.

If you want to make a cross language protocol that most languages can support, OOP concepts will need to be first class citizens.

Should I post the xkcd? Here goes: https://xkcd.com/927/

That's a grand idea, but there's good reasons to go to this effort for just C and C++.

There's a whole lot of code written in both; their toolchains are highly portable; for a long time they were seen as the only languages that gave programmers the ability to write high-performing code; and they are both unsafe by default.

Providing seamless bindings with a safe-by-default language like Swift is a way to move the whole systems programming field forward to a safe language, while maintaining compatibility with the enormous existing code base in the world.

Sure, but Swift is just one of many modern and (less modern) high-level languages, it's not just about connecting Swift to other languages than C or C++. There's also D, Rust, Nim, Zig, Python, Java, C#, even Javascript (when calling WASM modules), etc... those would all benefit from a more modern (and standardized!) approach to language bindings (even if the target languages would only be C or C++, but why stop there).

Apart from that "grand idea" I think a more realistic real-world solution for the C++ => Swift situation is to simply wrap the C++ API into a C API first, and then create a Swift API on top of the C API, instead of trying to somehow translate the C++ API directly to Swift. E.g. every C++ library should come with an optional C-API to simplify language bindings, or even just compiling that library into a DLL.

If we focus just on the programming cases where C++ has been heavily used, you can narrow that list down to D, Rust, Nim, and Zig. Dynamic languages or those with a heavy runtime like Python, Java, C#, and JavaScript are substantially different use cases.

Among those remaining, Swift has substantially more corporate backing, what with Apple relying on it for their massively successful iOS platform and Google trying to use it for TensorFlow.

C wrapper for C++ APIs has long been standard practice, but adds a substantial maintenance burden to C++ library authors and does not address semantics in the host language, which is a major part of this proposal. The whole proposal is based around the idea of seamless Swift to C++ interoperability. C++ has a lot of higher-level concepts that potentially can translate to similar higher-level concepts in other languages; round-tripping through a C API loses all of that.

To be clear, I like your idea - it's a grand one, and potentially solves the N+M problem once and for all. The real stumbling block for it is the C++ ABI and sheer mass of existing C++ code.

> E.g. every C++ library should come with an optional C-API to simplify language bindings, or even just compiling that library into a DLL.

hard to do without incurring additional dynamic allocations though. e.g. how would you make, say, an API that wraps construction / doing stuff / destruction of something that'd look like this :

    namespace mylib { 
    struct node { 
      std::vector<node> children;
      std::optional<float> value;
    }; 
    }

in C you'd likely have to have something that looks like

    mylib_node* mylib_node_alloc(void);
    void mylib_node_free(mylib_node*);

    // and then... 
    float* mylib_node_value(mylib_node*); // this ? 
    float mylib_node_value(mylib_node*, int* ok); // or this ? 

    void mylib_node_set_value(mylib_node*, float value, int set); // then there's this ?
    void mylib_node_set_value(mylib_node*, float* value); // or this ?
    void mylib_node_unset_value(mylib_node*); // or maybe this ?

and all this cannot really be done without a dynamic allocation (which hopefully looks terrible to everyone), unless you fancy doing ugly ugly ugly stuff like

    struct mylib_node { char storage[sizeof c++ struct that was computed manually)]; };

so anything that'd allow other languages to understand at least C++ vocabulary would be a great boon in terms of perf, etc

You can do something like

    // Or replace with something like a Strong Type; https://www.fluentcpp.com/2017/05/05/news-strong-types-are-free/
    using NodeHandle = int;

    struct OptFloat { float Value; bool bInitialized = false; }; 

    NodeHandle CreateNode(NodeHandle* Children, int NumChildren, OptFloat Value);
    void DestroyNode(NodeHandle ToDestroy);

    void SetValue(NodeHandle ToSet,  OptFloat  NewValue);

Aside from that, you do raise a good point. If you implemented your C api over your implementation of node, you would be in for a bad time as the owning std::vector<node> won't keep valid your pointers valid.. This isn't suposed to be ragging on your code, it's really to point out the pitfalls of manual APIs.

> struct mylib_node { char storage[sizeof c++ struct that was computed manually)]; };

You can do;

    // IN C api, fill this in manually
    #define mylib_storagsize = ?

    struct mylib_node {char storage[MYLIB_STORAGESIZE]; };
    // or;
    struct mylib_node {void* storage; }
    // in node.cpp - the underlying C++ abstraction;
    static_assert(sizeof(mylib::node) == MYLIB_STORAGESIZE);

It's only _slightly_ better..

I think LLVM is too low level for accomplishing the goal of generating a Swift-like API for calling foreign code.

Also, C, Objective-C and C++ cover a large part of the need (certainly when looking at Apple’s code base for their OS)

I also think any other interoperability standard for native code that works well for a broad range of languages is a pipe dream. There simply are way too many language idioms. Even for managed runtimes, you get lots of friction, for example between languages with mutable state and languages without.

> I think LLVM is too low level for accomplishing...

What I mean with "LLVM for language interoperability" is not using LLVM itself (although it could be part of the wider LLVM project), but the idea of LLVM. LLVM decoupled language frontends and code-generation backends by defining a common intermediate layer. Suddenly it was possible for individuals to create new languages that automatically run on all CPUs and ABIs supported by LLVM. What's missing for those small new languages is the same effect for connecting those languages to all the other languages that already exist. Doing this for each other language is impossible for small teams. Doing this for a single common interface specification on the other hand is fairly trivial.

Such an "hourglass shape" would also make sense for language interoperability. But instead of having a common intermediate bytecode representation, have a common intermediate API-definition + ABI specification (somewhat similar to existing IDLs, but without a separate IDL-language that must be manually maintained, but instead automatically derived from the "native"-language interfaces.

Also, it doesn't need nor should support all sorts of special language idioms, just a basic common set of "interface features". It would be very close to what we already have with the C API/ABI interoperability, just flavoured with additional annotations for some common higher-level concepts that languages have agreed on since C was invented (e.g. different types of references: unrestricted pointer, non-nullable pointer, ranged pointer, ownership annotations, etc...).

Once you have such a common set of annotations, languages can create more idiomatic (not perfectly idiomatic) API bindings automatically. It doesn't have to be perfect, just an evolution of the current state (basically "rich C-APIs" instead of "plain C-APIs"). And plain old C would still be part of the language mix that's supported, it would just ignore most of the high-level annotations.

I think that path gets you to the JVM or something similar.

Let’s say you use library A, get a string out of it, and want to pass that to a function in library B. That will only work if both libraries have the same idea about what a string is. The same argument applies to arrays, hash tables, etc.

Similarly, the wish to be able to have a serialization library that works across languages will add restrictions.

Also, do you support bare functions? If so, you’ll have to tweak things for Java. Do you support function overloading, and if so, how? Do you have multimethods? If not, you’re putting languages that do at a disadvantage. If you do, languages that don’t will be problematic.

The only examples I can think of where that works reasonably well are C/C++/Objective-C (with the C ABI as the intermediate), Java/scala (but note that scala was designed to work on the JVM, in the same way C++ and Objective-C was designed to work ‘on’ C), and C#/F# (again, F# was designed for .NET)

Swift is designed to work with Objective-C and, because of it, with C and, to a lesser extent, C++. This tries to make that a larger intent.

> That way every language only needs to provide a connection to an intermediate specification, and it can connect to any language supporting that same specification.

The intermediate specification would be another language, so it would have the same impedance mismatch as the other N languages do.

You have memory management issues, OOP inheritance issues, semantic differences. Just trying to do a interoperability layer for the different versions of character strings would be incredibly hard.

Microsoft did something similar to this with CLI, but most people will note that JScript.Net and VB.Net are really new languages that just are optimized for porting Javascript and Visual Basic code over. They also defined a proprietary extension to C++ that allowed it to deal with managed objects so that it could interoperate with CLI code - but this did not aim to expose unmodified C++ code. Rather, it allowed you to use proprietary extensions to manually build bridges between the two.

> The intermediate specification would be another language, so it would have the same impedance mismatch as the other N languages do.

Yes, but even though this is a limitation, every language only needs to care about connecting to this one intermediate "language". This is also why going through a C API already works quite well.

Also it's interesting to note that Microsoft's latest version of the WinRT APIs are providing a standard C++17 wrapper on top of COM APIs and don't require C++ language extensions anymore.

To the anger of Windows devs being told to wait for ISO C++23 or even ISO C++26, hoping that the standard gets the relevant papers for C++/WinRT VS tooling to catch up with C++/CX from 2012.

Right now it is hardly better than ATL/WRL, but so is life when Windows team wins its political battles with DevTools.

That already exists, and it’s called the „platforms C ABI“.

Every language that supports C FFI can talk with any other language that also supports it.

All these other efforts are just really higher level wrappers around that, but given that there are a million tools for generating the glue code to bind APIs using C for most languages in use it is hard for me to see the value here.

No, you're just wrong. What parent means is something like COM or GObject. The C ABI is too limited and still requires you to build things on top of it.

It's kinda like saying all you need is bytes. From the perspective of the kernel this might be fine because it isn't supposed to prescribe a particular method but user processes must still agree on a character encoding or a protocol they are going to run on top of those bytes.

For the platforms written in C.

> Wouldn't it be better to start working on an "LLVM for language interoperability"?

> trying to create the "one universal language to rule them all"

I feel this rebuttal of your proposal should be made, but I'm not sure how to articulate it more convincingly than you did.

To be fair though, for people developing for iOS it's a disproportionately huge point in the "huge M:N matrix".

My experience is that grand solutions can be attractive in theory but awful in reality.