How to Write a Rust Syntax Extension
brodoyouevencode.com
brodoyouevencode.com
If I have a subtle metaprogramming problem that I'm going to solve with the C preprocessor, I can code it and document it right there in the same source where it's used. Because that's the natural place for it. The metacode is the fundamental thing I'm writing, and it belongs in the "code" files for the rest of the project.
These Rust things are implemented as modules. I need to keep them separate. I need to build them separately. That breaks discoverability really badly. If a future maintainer needs to figure out what my strange macros do, they just look at them. If they get confused by my strange syntax extensions, they need to figure out where some other crate is coming from.
Regardless of other concerns about the mechanics, I think this is going to put a really serious damper on this kind of code.
Quite honestly, I'm looking at this and seriously thinking that more pedestrian options (python scripts to emit or preprocess rust code, for example) might be simpler and cleaner in practice.
Syntax extensions are more like plugins for the Rust compiler.
Thinking about this as a "compiler plugin" is exactly what I think I'm getting at: if I need to implement some sort of DSL feature for my code, I want to do it as part of my code and not have to deploy it as a plugin to the compiler.
Cargo.toml:
[dependencies.plugins]
path = "../plugins"
#![plugin(plugins)]
Generics are Rust's answer to most template metaprogramming.
> if I need to implement some sort of DSL feature for my code, I want to do it as part of my code and not have to deploy it as a plugin to the compiler.
The problem with that is that, in a cross-compilation scenario, the environment that the compiler runs in may be completely different from the environment that your code is deployed in. Different symbols may be available, some libraries may or may not be there or work differently, etc. So you really need it to be separate modules.
(Edit, for clarity: the C preprocessor and C++ template languages are well-specified interpreted languages executed at compile time. Rust's attempt here is poorly specified and coupled closely to the runtime environment of the compiler binary. And that choice has leaked into the API presented to the programmer. And that's bad.)
And regarding templates/generics: I'm not talking about type calculus stuff, but more like the kind of code you see in Boost's lambda or serialization libraries (which to be fair I think are terrible libraries personally -- IMHO this kind of code doesn't belong in libraries at all, which is my point). I don't think Rust generics (to their credit) can do that.
Two of your examples have been template metaprogramming and cpp, both of which are very limited programming environments (technically Turing complete, but both are Turing tarpits). When you actually want a full programming environment, you always hit the cross-compilation issue. It's just fundamental to the problem. But the advantages of a having a full programming environment available, when you need that power, outweigh the downsides of having to write in a separate module.
The other example you gave is writing a Python script, which has all the same problems of writing in a separate module, except now you're writing in a different language in addition to writing in a separate module.
Sometimes you don't need the power of a real programming environment, of course. That's why the macro system, which does not expose a full programming environment, exists in Rust. But C++ templates and (especially!) cpp weren't designed as programming environments either. They're pretty terrible programming environments. If you want that kind of thing, use the macro system. It's technically Turing complete too in just the same way as templates are. But we also expose a full programming environment, if you want it.
I'm mystified by how offering a feature that extends the existing state of the art in addition to offering a feature that reflects the state of the art—the macro system—could possibly be somehow "doing it wrong".
That's a total strawman (seriously: every time I see you deal with a criticism you pull out this flamage toolkit; please stop). All that stuff in the first half of your sentence? That's the stuff you're doing right.
What you're doing wrong is that I can't write that inline in the code and need to write it as essentially an extension library for the compiler.
I don't think it's such a strange request that my DSL implementation look like it does in C or Scheme, e.g. "[Define some quirky DSL hackery] [Implement a bunch of code using it]". I'm likewise mystified why you think that's not a useful requirement or a shortcoming (even a minor one -- you're literally refusing to engage me on this) of your design.
The example code in the linked article is simply ugly. Ugly APIs are bad, even (especially) for subtle features.
This is the use case for macros, and Rust macros are at least as good as C macros for my common use cases. It's true they are more limited than Scheme macros, and also fairly different from C++ template metaprogramming.
Rust macros aren't perfect, and they'll certainly improve as the language matures.
Also, for cases where a compiler plugin is needed, I'm not sure why having it in a separate file is such a deal-breaker. Opening multiple editor tabs/windows/frames has never seemed that annoying to me.
> The example code in the linked article is simply ugly. Ugly APIs are bad, even (especially) for subtle features.
Yeah, Rust syntax extensions at this point are basically just reaching into the guts of the compiler and using APIs that were never meant to be public. That's why they're not part of Rust 1.0, and won't become part of the stable language any time in the near future. Right now I see them partly as a way of prototyping compiler/language features. Hopefully in the long term they will grow into (or be replaced by) something with a stable, well-designed API that still exposes most or all of the current power.
By the way, Servo uses Rust macros, Rust syntax extensions, and two different Python script for code generation. (One is a fork of Gecko's WebIDL parser for generating DOM bindings; the other pre-processes a source file writen as a Mako template.)
It seems that your position is "well, you won't get it so shut up about it". That seems rather uncompelling too. :)
> I don't think it's such a strange request that my DSL implementation look like it does in C or Scheme, e.g. "[Define some quirky DSL hackery] [Implement a bunch of code using it]". I'm likewise mystified why you think that's not a useful requirement or a shortcoming (even a minor one -- you're literally refusing to engage me on this) of your design.
In Rust, you can implement any DSL you could implement in C precisely as you would in C, using Rust's macro system. You define a macro inline in your file, and then you can use it right there in your file. No separate module is necessary. Rust's macro system is pretty much strictly more powerful than that of C, so anything you can do in C macros you can do just as easily in Rust.
When you want more powerful syntax extension features than what Rust's macros—or the C preprocessor—could provide, then you reach for the compiler plugins.
When it comes to Scheme-like macros, where the language in its entirety can be used at compile time, that's where the cross-compilation story makes things more interesting. Rust's syntax extensions are just like Scheme macros in that the entire language is available to you. In Scheme, however, your compile-time and runtime environment can generally be considered to be one and the same, which makes cross-compiling issues a lot easier to deal with. Anything available at compile time is also available at runtime, so you can do things like write modules that export functions available in arbitrary phases. In Rust, because of things like cross-compilation, this is not true: your compile time and runtime environment can be totally incompatible. You can't just write a helper function inline in your code and expect it to be usable both at compile time and at runtime, because its compilation environment (for example, the symbols it references) may or may not exist depending on the architecture.
I don't think I'm "refusing to engage"; I'm just saying that we considered everything you're talking about already. The Schemers on the team had exactly your complaint :) But we tried to make a simple implementation that did what Scheme did and ran headfirst into the cross-compilation issue, pushing us toward a more limited approach. It's quite possibly doable to implement syntax extensions in such a way as to allow for the full generality of the language at compile time while still supporting cross-compilation; SBCL can cross-compile itself, for example. But it's a lot of work, and all of the Lisps and Schemes that can do something similar have a huge amount of infrastructure to allow them to do so. (Read the SBCL cross-compilation paper, for example.) Certainly no implementations of the main compiled languages I'm aware of allow the full generality of the language to be embedded within itself without some sort of module boundary: C++ and D compilers use an interpreter for a subset of the language to evaluate constexpr (and the reason why the full language is not supported is, essentially, exactly this). Given that Rust is hardly unambitious to begin with, I'm OK with the fact that it doesn't solve this problem yet.
These sorts of macros work very well for large classes of things one would do in with those C/C++, and more beyond that too. Syntax extensions are particularly good at things that not possible without massive hacks in either (although, again, this is somewhat changing with constexpr in C++14 etc).
On the point of being inline, yes, it would be helpful sometimes, but often syntax extensions are "standalone": useful functionality that makes sense to be in its own crate (e.g. for publishing to crates.io), especially while they are unstable. And cargo makes it so easy to use other crates (even local ones, even plugins) that it barely matters where it is defined in terms of tooling/infrastructure.
I think this is a typical example for the huge verbosity of Rust. Yes, you get safe bare metal access but for the price of a lot of coding effort for _every_ application.
For comparison watch how easily syntax extensions are done in Nim. For instance,
template times(x: expr, y: stmt): stmt =
for i in 1..x:
y
10.times:
echo "Hello World"
Furthermore, your example is misguided. Syntax extensions are not for trivial rewriting, that's the purpose of macros. Syntax extensions are for doing things like expanding a regex into a finite state machine at compile-time, or providing custom extensions to Rust's type system. In terms of tools, they are a 40-foot industrial lathe which can do anything, but is overkill for the vast majority of tasks.
Nim provides templates and macros for that. Templates are for rewriting while macros can create arbitrary AST nodes.
http://blog.ldlework.com/2015/05/01/a-cursory-look-at-meta-p...
Also, this domain makes it really hard to take the content seriously...
That said, still glad to see posts like this.
Also, thank you for an awesome article. Any plans to try to get this document added to Rust documentation? Even if it's too unstable for the standard book, it'd be nice to have around as part of the documentation.
Java has had this feature for about 10 years now. The standardized API (required by all conformant compilers) defines only reading a partial AST (just declaration) and generating new source files, but javac supports full AST transformations.
Two famous Java projects making use of javac AST transformations are the good-old Lombok project and Checker, Java 8's new pluggable type systems.