mrustc: In-progress alternative Rust compiler (to C) | Hacker News Reader

mrustc: In-progress alternative Rust compiler (to C)

github.com

80 pointsmy12362 comments

The interesting thing to me is that this does not fully valid Rust code, e.g. not checking for borrow safety. Which would normally make no damn sense for a Rust compiler, but actually makes sense for what this particular project is trying to achieve.

This is not a compiler for developing Rust programs. This is a compiler to let platforms unsupported by Rust/LLVM build rust programs from source. So it takes a Rust program which already exists, assumes it can be compiled by rustc, and translates it to C so it can be handed off to your local compiler. Skipping the burrow checker probably lets this execute in environments which are more constrained than what rustc can fit into.

Of course there's a question of how much of Rust's safety survive translation into C. Certainly some. E.g. the emitted program would include bounds checks where not specifically elided in Rust, and references are statically guaranteed to be non-null before being lowered into pointers. But there could be gaps, especially in cases where Rust semantics mismatch with C in ways that LLVM hasn't exposed yet.

> But there could be gaps, especially in cases where Rust semantics mismatch with C in ways that LLVM hasn't exposed yet.

Could you elaborate on what potential mismatches there currently are between the semantics of Rust and C? Where particularly do you think there might be issues/dragons hiding in the semantics of a translation?

Integer overflows (signed overflows defined to wrap or panic in rust, are undefined behavior in C).

Float integer conversions (overflows are defined to saturate in rust, are undefined behavior in C).

Maybe things like how pointer casts are treated in correct (but unsafe) rust vs C. Or similarly in transmute. Generally it wouldn't surprise me if rust was standardizing a memory model that was subtly different from C.

I'm sure the list goes on, but "that sort of thing".

Pretty sure rust wraps on overflow, not saturates, but it's not hard to get that behavior from most C compilers if desired.

float to integer saturates in Rust. `99999.0 as u8` gives 255. `(char)99999.0` is UB in C and LLVM. Rust had to add explicit clamping code to float casts, because LLVM compiled nasal demons otherwise: https://github.com/rust-lang/rust/issues/10184

Rust also has different aliasing rules. Stricter than C for references, but more relaxed than C for raw pointers.

Clamping floats is interesting, but seems pretty niche.

Would have been nice to provide notice of your edits.

I'm not sure if I edited at all, if I did edit it was within a minute or two of my original post.

Edit: Now that I am coming back and reading this though I am going to make an edit, and like normal for substantive edits I'm going to mark it clearly...

Edit2: Or I can't edit, so let's put it here:

A better example than any of the above might be infinite loops, `while (true) {}` (or any similar loop without side effects) is undefined behavior in C, it's defined behavior in rust.

Unfortunate timing.

> This is not a compiler for developing Rust programs. This is a compiler to let platforms unsupported by Rust/LLVM build rust programs from source. So it takes a Rust program which already exists, assumes it can be compiled by rustc, and translates it to C so it can be handed off to your local compiler. Skipping the burrow checker probably lets this execute in environments which are more constrained than what rustc can fit into.

I would assume that borrow checking uses far less memory than the actual compilation after the compiler has proven the program to be in concordance with the borrow rules.

It more so seems that it is something they did not bother to put too much effort into, as it wasn't necessary, than a legitimate way to reduce a compiler's memory footprint

To me the question it raises is - does the rust borrow checker need to be part of the compiler or could it be separated into a standalone component? Is the borrow checker architecture-agnostic?

It is architecture-agnostic.

As currently implemented in rustc, it operates on an intermediate representation that has already had name resolution and some simplification done. As such, i think it would be rather hard to separate from the rest of the compiler.

For the first one, definitely could be split. You can already run that part without code generation - https://doc.rust-lang.org/cargo/commands/cargo-check.html

It would be a nice addition in use similar to the auto-complete function - when you write a line of code that uses a value the breaks the tiles, it highlights the code right there.

That alone would make rust a much easier language for newbies.

The borrow checker is actually fairly simple to implement, if you have the types. To deduce types, you need to parse the grammar, understand the control flow, and resolve any builtins. But you don’t need to emit any further representation.

There is also a WIP rust gcc frontend: https://github.com/Rust-GCC/gccrs

I wonder how these two projects compare.

Oh, nice! I didn't realize that development was still happening for that, I thought it had stopped around Rust 1.20. Glad to see that it's still being developed, stuff like this will definitely be important for bootstrapping/"trusting trust".

It’s strange for me why anybody would reimplement a Rust compiler in C++ instead of Rust, but people are free to do what they want :)

Part of the point is allowing bootstrapping a rust toolchain from source on systems that don't have rust yet.

Can't you just cross compile? Wouldn't you have to do that anyway for some parts of the system? What's the point of bootstrapping?

When I say "systems", I should have been a little more specific.

There's a bunch of corporate/.gov envs that don't allow binaries to enter their system. All new code has to be compiled from source, on the target system (or at least on that side of the process firewall, it might be an airgapped cluster that can share code). They have C and C++ compilers that they've been compiling from source since the dawn of time, but they don't have a rust compiler. This gives them a mechanism to begin using rust within their process.

Wow I see. Could they not use the initial Rust compilers (and however those were bootstrapped? I have no idea how) to get a Rust compiler going? I'm guessing it's just too painful?

Bootstrapping rust is really, really painful as is. The rust compiler itself tends to require a recent version of the rust compiler to work. So to bootstrap from the original versions of rustc (which were written in OCaml) you'd have to incrementally step up really slowly.

Being able to target a pretty recent version of Rust with a compiler written in C would be so, so useful for these purposes.

Certainly that could be automated quite easily.

I would assume that on such compilation farms that these systems generally use, this could be done very quickly.

Perhaps there would be an interest for the Rustc team to provide a recursive automated setup that is capable of compiling the latest Rustc from OCaml, and finally from C as well since one must follow a similar process with OCaml.

The automation for this is actually trickier than it would first appear: especially with the earlier snapshots, a number of patches are needed to compile the LLVM interfaces written in C++. Also, due to linking issues, some snapshots require a stage2 compiler, while others can only use a stage1 compiler. I agree that it would be trivial once you have the chain in place, but that chain can only be built through trial and error.

And no one has done this before?

Methinks that building this chain by trial and error is rather trivial compared to the actual work in building a compiler.

We are discussing mrustc here. The reason it hasn't been done yet is that nearly everyone interested in this agrees mrustc is a better bootstrapping path than starting from the last OCaml version.

That it's a better bootstrapping process for the user has no relationship with the ease of development for the developer.

Typically, they are inversely correlated.

Guix performed the bootstrap from mrustc 0.8 for rust 1.19 all the way to rust 1.28 in 2018: https://guix.gnu.org/en/blog/2018/bootstrapping-rust/

Since them they've bootstrapped this all the way to 1.50: https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages...

The work done here is non trivial, this chain of versions are required for the final binary to be reproducible.

And when you port to a new CPU architecture you're going to backport those changes to 200 obsolete versions as well?

Yeah, rustc dogfoods new features, so it's _very_ painful. It'd probably be close to 100 or so compilers to go back to the ocaml that it started from.

I've actually been trying this for real. Right now, I'm at the 82-compiler mark, including the ~6 LLVM versions needed. For a sense of scale, the most recent compiler in this chain is the 2011-12-07 snapshot, which is still older than Rust 0.1 (2012-01-20). In most cases, additional compilers are needed only to resolve syntax errors, but there are occasional LLVM-related segfaults that create most of my headaches.

Is your work publicly available somewhere?

The scripts are still pretty messy right now. I'll probably clean them up a bit and upload them once I reach 0.1.

> I'm guessing it's just too painful?

Indeed. Although it is being worked on, the whole bootstrap process of rustc is currently a major hassle, requiring to start from the oldest CaML versions of the compiler up to the most recent ones.

The earliest versions of the rust compiler were written in Ocaml, which is also unlikely to have been built in a trusted way by these organizations. mrustc exists to be a way to bootstrap trusted builds of rustc.

But there are no binaries being copied. They allow hand-written C to be copied, but not machine-generated C?

Without mrustc you'd need to trust the rustc binary, or build around a hundred versions of rust to go back to a rustc that itself didn't depend on rust.

Do all these entities operate independently? Is there no internal web of trust they could use to distribute rebuilt binaries?

Yeah, they operate independently. No new binaries, period.

Which makes sense: compartmentalise. Otherwise, a failure of one domain would compromise all the others.

I hope there's some automated assistance for scanning inbound source code too. Imagine reviewing everything, line by line, millions of them.

How did they get the original C compiler there then without allowing binaries? Did they originally write one in assembler?

I remember reading an interesting thought experiment that traced history that investigated what would happen if Dennis Ritchie had put malicious code in the first C compiler that was designed to detect whether the compiler compiled a compiler, and then copied the malicious code into it.

It concluded that tracing the history, that if this were to have happened, then GCC and Clang and many other programming languages would have said malicious code in their compilers that do not show up in the source code with none the wiser.

Epoch creation is covered by a totally different set of process rules, for the reasons you've stated. These rules are mainly intended to make it much harder for someone to gain access to the system if they weren't there when the system initially was blocked off from the world.

If understand Dennis Ritchie's explanation [1] correctly, he started with a compiler for the B language, itself written in B, and modified it to accept a somewhat different language, which he called C.

Before the B compiler written in B, there was a B compiler written in something else. I couldn't figure out what from that document.

[1] https://www.bell-labs.com/usr/dmr/www/chist.html

Couldn’t they run mrustc on the rust source code on another machine to generate C code for rustc, then copy the C code to their internal systems and compile that into a working rust compiler?

Or would that not count as “compiling from source”?

I believe that’s exactly what they were suggesting :)

I see, thanks for the explanation, now the project makes a lot of sense :)

I guess you didn't want to advertise the real reason of the project.

One way to trust your computer system more is to have very few binary dependencies. Ie the goal is to have some source files and hopefully if you start with (eg) a trusted C compiler and linker you can work your way up to the whole system with deterministic results. This is supposed to be a way to prove that the compiler wasn’t doing anything funky like injecting some vulnerability into the code it compiled and injecting that vulnerability injection into any compiler it compiled.

I thought this was already possible. The first Rust compiler is written in OCaml. It should be possible to do bootstrapping with that.

Of course, but AFAIU this massively shortens the chain. Newer versions of rustc up the minimum compiler version required for building all the time.

Unfortunately rustc uses rust features almost immediately (I think even features introduced in the immediately preceding version?), so you get a bootstrap chain, and if you start with the compiler written in OCaml you now need to build every single rustc version ever released.

This takes very long and is error-prone, so having a compiler that can build even an intermediate version is already a big help.

See e.g. https://guix.gnu.org/blog/2018/bootstrapping-rust/

It's unfortunate that Rust has adopted development policies that are so hostile towards bootstrapping and cross-compiling rustc. The fact that most of the people cheerleading for Rust the ecosystem don't even see this as a problem is enough to convince me to steer clear of anything involving Rust the language.

You don't have a real language until there is a stable specification and multiple independent implementations. Until then it's just an experimental toy.

You can't build GCC on MSVC or vis versa.

And people do see the pain here as a problem, that's why the project in this very thread exists. I think most in the Rust community see mrustc and the pressure that an independent implementation brings process wise as a very good thing for the language.

At least some years ago, you could build GCC on MSVC just fine. (Did it for GCC 3.3 and 4.2 on Interix way back then)

It should probably be still functional, or easily fixed if not.

Ah yes! The experimental toy languages without specification like Python.

I'm sorry, what are PEPs, again?

PEPs are specification fragments. But it's far from a complete python language specification.

CPython is the only viable Python interpreter, and the Python language specification is "whatever CPython does". In my humble opinion this disqualifies Python from consideration as a language.

That doesn't mean Python isn't useful! It is in fact a wildly successful tool that has spawned a large ecosystem of packages and a diverse group of users. But it unfortunately does not qualify as a true language, for it lacks the necessary ingredients: a formal specification of the language and more than one implementation of said specification.

>is enough to convince me to steer clear of anything involving Rust the language.

This has a "I will cancel my subscription" letters to the editor feel, as if anyone cares :-)

I mean, there are pros and cons to any language and ecosystem. "I found some fault in bootstrapping approach and will condemn the whole language/effort for it" is a hardly a reasonable argument...

As if regular devs often have a need to bootstrap from scratch themselves...

Since then they've bootstrapped their way from 1.28 all the way to 1.50: https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages...

You mean just like LLVM?

Strange? The whole point of this is easy bootstrapping, so it's anything but strange, it's the main point...

Reply on news.ycombinator.com