Hacker News Reader

Comment by kibwen | Hacker News Reader

I was about to say, don't bother submitting a patch to fix this, as last week rustc completely moved away from make, deleting all its makefiles in the process: https://github.com/rust-lang/rust/pull/39431

I'm curious how OP even ended up trawling old versions of rustc's makefiles from 2014. I bet there's an interesting story there. :P

View on HN

I blame this twitter thread :P

https://twitter.com/Gankro/status/831955401109360640

The thing I actually kind of think is amazing is that someone said "This isn't working... why don't I try putting in a $(nop) here...?"

I don't think using noops to get the parser to co-operate is that strange at all, especially not something like a makefile where whitespace is significant. It's the whole point of the `pass` noop in python: the Python grammar expects there to be something inside a loop, so if you want an empty one, you have to throw a noop in there.

> the Python grammar expects there to be something inside a loop

I think this is the underlying issue - the "well, it's a list, so there's got to be something inside the list too" mindset, which makes sense in the real world but is stupid in programming. (Some bad memories at old Visual Basic times come to mind too)

Maybe there was something else there before, and it stopped working when they removed it, so they tried adding back the most minimal thing they could think of as a sanity check.

"Why doesn't this work?". Add debug statement. "Huh, now this works". Take out debug statement. "Huh, now it doesn't work". Try no-op. Table flip. Commit.

I love Heisenbugs because I have seen one in the wild. It's the most absurd thing ever.

Maybe they even knew that white space would work. However some people (including me) wouldn't even look twice before they remove white space. An explicit no-op on the other hand would get my attention.

I have no regrets. Everyone should get to know the fear and joy of this Makefile.

That was it indeed :)

So they switched from make to something called rustbuild, it appears. A quick search didn't turn up any details on rustbuild for me. Is it something very rust specific, or could I use it for other projects?

It's specific to rustc, to satisfy the unique demands of bootstrapping compilers. Unlike the makefiles it leverages Cargo as much as it can, which means that the compiler will gradually become less insular and will also benefit from future Cargo development.

Right now the non-Cargo bits are written in Python (which was already required to build LLVM, so this doesn't add any new dependencies), but those bits could pretty much be written in anything, if needed.

rustbuild is actually pure-Rust. The entire Rust build is written in Rust (Rust is bootstrapped all the way down!). The python scripts are a slim wrapper that downloads the bootstrap compiler to get into the Rust world.

Anyone know how Debian is handling this? Debian packages are not allowed to access the network on build, and (normally) are not allowed to depend on non-Debian binaries.

rustbuild can optionally build from vendored (in-tree) source of all cargo dependencies, where it does not touch the network, and that's how we expect packagers to build Rust. I believe Debian has their own bootstrap lineage where they bootstrap from Debian's own prior builds, and rustbuild supports this mode; but they also have a fallback for rebootstrapping, with the corresponding policy exceptions.

If you are interested, here's a draft Debian Rust packaging policy: https://internals.rust-lang.org/t/debian-rust-packaging-poli.... It is not about packaging Rust itself though but the challenges of packaging software written in Rust.

Is any version of rust still buildable from the previous OCaml version? Could it still be bootstrapped from that point?

I think it's bootstrappable, but I haven't tried:

rustboot, the OCaml Rust implementation, was removed in 6997adf7 on 13 May 2011. It looks like it was fairly actively maintained until then.

A file named src/snapshots.txt showed up in the repository in cb53065a on 2 May 2011, and remained there until 02538d46 on 13 April 2016. During that time, rustc was buildable from specific snapshots of previous versions of rustc, which were listed by both source commit and hashes of binary compilers.

From 13 April 2016 onwards, stable rustc compiles on previous stable Rust, and specific versions are listed in src/stage0.txt in a similar fashion.

So, you can probably take the last version of rustboot (it didn't change between 2 and 13 May 2011), compile that with some version of OCaml, then go from 2011 to 2016 by repeatedly building the newest snapshot possible with your current snapshot, then build stable Rust 1.10 onwards, thereby reducing it to the previously solved problem of verifying that Ken Thompson has not compromised your OCaml compiler.

They have to be done in order. Only the first version that was in Rust could be compiled from the OCaml compiler. Then each one in series only builds with the specific parent compiler.

> (normally) are not allowed to depend on non-Debian binaries.

It turns out that Debian is more permissive than Fedora when it comes to bootstrapping compilers with out-of-archive compilers. Fedora has an actual policy and it allows one-time out-of-archive bootstrapping.

It is specific to the compiler.

https://github.com/rust-lang/rust/tree/master/src/bootstrap

It basically consists of:

1. A python script to make sure you have the correct environment, and that kicks off cargo to compile the program used in step 2. ("rustbuild" proper.)

2. A Rust program that manages the full build (remember, Rust is bootstrapped, so there's three stages...)

3. That build kicks off a number of "cargo build"s for all the components and glues the final result together.

For any project written in Rust, pure Cargo should be enough.

I've always been a bit confused about Rust's need for so many build stages. The project I work on (crystal) only requires you to install a working build of the latest compiler release to build. (the makefile is pretty simple). As far as I know, rust can't bootstrap without downloading a rust release, so can you explain why rust has compiler stages.

So, there's three stages:

0. This stage is the pre-built compiler

1. This stage is a compiler built with the compiler from stage 0

2. This stage is a compiler built from stage 1.

If the process worked, then stage 1 and stage 2's output should be identical. You can skip stage 2 if you want to skip that verification, and that sounds like what Crystal does.

I always mess this up, but here's something from memory: I believe there's ABI things that motivate stage 2. The compiler built in stage 0 has historically had special ifdefs to deal with "features that don't exist in the old snapshot" (we aggressively adopt our own features in the compiler and stdlib, because that's a great way to test them). This implies it can potentially emit binaries with a different ABI from the compilers built in stage 1 and 2. So the compiler that's output from stage 1 can actually have a different ABI from the binaries it produces.

Normally this wouldn't matter, but Rust also supports compiler plugins written in Rust. These link directly to the compiler that compiles them. With the stage 1 compiler's quirky ABI, the plugins it compiles might not be able to interact with it.

The stage 2 compiler, on the other hand, uses the same set of ifdefs that the stage 1 compiler has. So it was built by a compiler with the same output-ABI as itself, and so plugins have the same ABI.

So:

* snapshot: Built with snapshot ABI, produces snapshot ABI (snapshots are old stage2 compilers)

* stage0-output: built with snapshot ABI, emits stage0 ABI.

* stage1-output: built with stage0 ABI, emits not(stage0) ABI.

* stage2-output: built with not(stage0) ABI, emits not(stage0) ABI.

Ah yes, crystal doesn't have compiler plugins, which simplifies things a bit. Say, if you specified that you had to be able to build rust master with the latest nightly release, could you use only 2 stages (latest nightly, built master)?

I'm currently working on crystal's CI infrastructure so that we can have nightly crystal builds (+ CI) for all architectures we support. Currently we release features incrementally such that the compiler can always be built with the latest release, and I'm wondering how much effect relaxing that constraint to the latest nighty would have on development speed.

Yeah, I was thinking of the more general case; gcc and such don't need to deal with this aspect, but I think you're right with Rust.

Additionally, if one is building a compiler for other targets (both a compiler that runs on another target, and just a simple cross-compiler), the makefiles/rustbuild need to make sure the appropriate things get cross-compiled in the appropriate stages. Managing this was a source of a lot of complexity in the old makefiles.

Is it possible to bootstrap from a compiler compiled from C, or another language? Or is there another way to ensure there aren't any sneaky compiler backdoors?

You could go back to the OCaml compiler. It would take about 1000 builds, each taking 3 or 4 hours.

Someone is working on an alternate compiler so we could try DDC, we'll see.

Sneaky backdoors already exist!

http://manishearth.github.io/blog/2016/12/02/reflections-on-...

(Of course, that's a proof of concept and not actually in the released binaries :P )

Reply on news.ycombinator.com