Hacker News Reader

Comment by RX14 | Hacker News Reader

I've always been a bit confused about Rust's need for so many build stages. The project I work on (crystal) only requires you to install a working build of the latest compiler release to build. (the makefile is pretty simple). As far as I know, rust can't bootstrap without downloading a rust release, so can you explain why rust has compiler stages.

View on HN

So, there's three stages:

0. This stage is the pre-built compiler

1. This stage is a compiler built with the compiler from stage 0

2. This stage is a compiler built from stage 1.

If the process worked, then stage 1 and stage 2's output should be identical. You can skip stage 2 if you want to skip that verification, and that sounds like what Crystal does.

Additionally, if one is building a compiler for other targets (both a compiler that runs on another target, and just a simple cross-compiler), the makefiles/rustbuild need to make sure the appropriate things get cross-compiled in the appropriate stages. Managing this was a source of a lot of complexity in the old makefiles.

I always mess this up, but here's something from memory: I believe there's ABI things that motivate stage 2. The compiler built in stage 0 has historically had special ifdefs to deal with "features that don't exist in the old snapshot" (we aggressively adopt our own features in the compiler and stdlib, because that's a great way to test them). This implies it can potentially emit binaries with a different ABI from the compilers built in stage 1 and 2. So the compiler that's output from stage 1 can actually have a different ABI from the binaries it produces.

Normally this wouldn't matter, but Rust also supports compiler plugins written in Rust. These link directly to the compiler that compiles them. With the stage 1 compiler's quirky ABI, the plugins it compiles might not be able to interact with it.

The stage 2 compiler, on the other hand, uses the same set of ifdefs that the stage 1 compiler has. So it was built by a compiler with the same output-ABI as itself, and so plugins have the same ABI.

So:

* snapshot: Built with snapshot ABI, produces snapshot ABI (snapshots are old stage2 compilers)

* stage0-output: built with snapshot ABI, emits stage0 ABI.

* stage1-output: built with stage0 ABI, emits not(stage0) ABI.

* stage2-output: built with not(stage0) ABI, emits not(stage0) ABI.

Yeah, I was thinking of the more general case; gcc and such don't need to deal with this aspect, but I think you're right with Rust.

Ah yes, crystal doesn't have compiler plugins, which simplifies things a bit. Say, if you specified that you had to be able to build rust master with the latest nightly release, could you use only 2 stages (latest nightly, built master)?

I'm currently working on crystal's CI infrastructure so that we can have nightly crystal builds (+ CI) for all architectures we support. Currently we release features incrementally such that the compiler can always be built with the latest release, and I'm wondering how much effect relaxing that constraint to the latest nighty would have on development speed.

Is it possible to bootstrap from a compiler compiled from C, or another language? Or is there another way to ensure there aren't any sneaky compiler backdoors?

You could go back to the OCaml compiler. It would take about 1000 builds, each taking 3 or 4 hours.

Someone is working on an alternate compiler so we could try DDC, we'll see.

Sneaky backdoors already exist!

http://manishearth.github.io/blog/2016/12/02/reflections-on-...

(Of course, that's a proof of concept and not actually in the released binaries :P )

Reply on news.ycombinator.com