>The limit being that the whole program, including all dependent crates need to compiled from source with the same compiler.

Which isn't really much of a problem given that the ecosystem is built around cargo compiling from source.