Closing the gap: cross-language LTO between Rust and C/C++

Now if only LLVM could feel it was important enough to make noalias work reliably.

I looked through the archives and only GHC/Haskell and LLILC have made it: http://blog.llvm.org/2010/05/glasgow-haskell-compiler-and-ll... http://blog.llvm.org/2015/04/llilc-llvm-based-compiler-for-d...

But I think the limitation is on developers willing to write a blog post rather than "serious" development. Excluding "LLVM Weekly" which has since moved to http://llvmweekly.org/, there's been less than 1 post per month, even though LLVM conferences have grown significantly in size.

You're right, but it still looks like a big improvement. It means that Firefox devs can write every new functionality in Rust, no matter how small it is.

LLVM bitcode is backwards compatible. It is however not forward compatible, so the linker needs to understand the newer bitcode format that clang and rustc use.

The long-term plan for Rust is to treat its standard library (almost) like any other crate, so it could be recompiled with your custom compiler settings if necessary.

However, libstd is by necessity tied quite closely to the compiler, it's one of the oldest and most fundamental parts of the stack, and there are tons of little complications around making it "just" a crate, so changes to libstd/rustc/Cargo necessary to make that possible will take longer.

In the meantime, changing one problematic compiler flag seems like a sensible solution.

The low-level tools team consistently reports bugs against the upstream projects. I have no doubt that they did so during the course of this project.

This isn't anything new. Rust has had to land workarounds for lots of LLVM issues in its history. For example, Rust had to stop using noalias on function parameters because LLVM miscompiled too many functions with it, as Rust can use it way more than C/C++ do and therefore it didn't receive much upstream test coverage.

It's just how it works out in practice, in my experience. LLVM is a large, fast moving target that's incredibly complex to understand, because it has a complex job. It resolves many issues for you when you're developing a compiler, but you are gifted issues in return. One of those is that understanding, diagnosing, and properly fixing problems can take a very large amount of work. It has bugs! I mean, LTO has historically been fragile for a single source language when enough code gets thrown at it, much less two languages!

Another is that compiler developers often don't have infinite amounts of time to sort out shenanigans like this when they come up. Users generally prefer the compiler to work, even if suboptimally, when compared to "not working", so there is some tension between things and how long they take. So landing workarounds in various ways -- sending patches upstream, using custom builds with bespoke patches, code generation workarounds -- all have to happen on a case by case basis. Many LLVM clients do this to varying degrees, and a lot of features like LTO start off fragile due to these kinds of things. Over time things will get tightened up, hopefully.

When I worked on GHC (admittedly several years ago now), the #1 class of problems for the LLVM backend were toolchain compatibility issues above all else, because we relied on the users to provision it. At the time it was nothing short of a nightmare -- various incompatible versions between various platforms causing compilation failures or outright miscompilation, requiring custom patches or backports at times (complicated by distros backporting -- or not backporting -- their own fixes), some platforms needed version X while others were better served by Y, flat-out code generation buts in GHC and LLVM, etc. It's all much better these days, and many features/fixes got landed upstream to make it all happen, but that's just how it works. Rust made several design choices with their packaging/LLVM use that we didn't that I think were the right ones, but I'm not surprised they've had a host of challenges of their own to address. TINSTAAFL.

All FFI in Rust requires `unsafe` so it's impossible to do this "without doing anything".

I think this is both possible and desirable. Basically, create a bridge between rustc and Clang, kind of like Swift's C importer, but far more complex in order to be able to do things like instantiate C++ templates on demand (perhaps even with Rust types as arguments!).

However, it would also be extremely hard; I don't think any programming language has ever created such a tight bridge to C++. The existing approaches I've seen are:

- Swig, rust-bindgen, etc.: Pass through auto-generated C ABI wrappers; support for generics is limited and requires declaring up front what types you want to instantiate them with.

- D: You can bind directly to C++ if you rewrite your C++ header in D, generics and all... including the implementations of all inline functions.

Both very limited, especially in template-happy modern C++.

The thorny issue is C++ templates, which are instantiated in a complicated way that depends on C++ syntax. You can't "just" compile and link Boost in (as you'd do with a C library). You have to compile a fragment of C++ source code for every line of Rust code that uses a Boost template, and you have to translate Rust code used in template arguments to valid C++ code, so that template substitutions will be valid.

This seems doable for simple cases, but libraries like Boost push templates to their limits, so they're not the easy cases.

You don't have to love Rust to still use it in specific narrow areas.