A Practical Intro to Macros in Rust | Hacker News Reader

A Practical Intro to Macros in Rust

danielkeep.github.io

188 pointsjxub24 comments

Note that this dates back to Rust 1.0, and although the macro system hasn't changed since then, plans are in the works to succeed this original system ("macros 1.0") with an improved one ("macros 2.0"), culminating in the deprecation of the original. The two systems won't differ that much, so the documentation will remain mostly the same and the transition should be very straightforward.

The impetus for this is that the macro system was unanimously considered the least mature feature of pre-1.0 Rust, but there was little will to delay Rust 1.0 even further solely for the sake of macros. Instead, some expedient band-aids were slapped over it during the run-up to 1.0, with the intent to eventually fix the whole thing properly when time allowed.

To wit, the main problems with macros 1.0 were that macro "importing" is an ugly hack that stands out like a sore thumb from how modules work in the rest of the language; in addition, despite supposedly being hygenic, there are some edge cases where hygiene does not hold. The new system will make the language overall more consistent and reliable (and might bring some minor syntactic improvements), with the obvious downside that making the macro system nicer to use means that people might actually start using it. :P

> plans are in the works to succeed this original system ("macros 1.0") with an improved one ("macros 2.0")

Do you have any pointers where this work happens and I can follow along. Searching the RFC's for 'macro' turns up too much noise. Do you know about a timeline?

While not directly what you are looking for, the release milestone predictions for rust is a fairly nice page IMHO to see current plans: https://internals.rust-lang.org/t/rust-release-milestone-pre...

For example, one can see that "Macro use and rexport" (https://github.com/rust-lang/rust/issues/35896) that will make macros a part of normal Rust naming and modules is planned for Rust 1.20 to be released end of August.

Actually I think that prediction is now inaccurate, as they in your link "Macro use and rexport" say that they want to postpone the macro changes until Macros 2.0 are ready as a whole.

Oops, thanks for pointing it out. It was a while since I read the discussion on "Macro use and rexport".

It is actually one feature I'm quite looking forward to, since I think proper naming for macros will make the whole eco-system simpler. Ah well, now we'll have to wait even longer.

Unfortunate, because metaprogramming is very important to code design. I won't want to abandon code just because v2.0 will get deprecated and shiny v3.0 will become the new standard next year. If Rust team cared about backward compatibility they could preserve old code by adding something like -std=rust1.0 which would not break code. Forcing people to rewrite any code(esp. complex macros) will not gain Rust any goodwill longterm.

https://github.com/rust-lang/rust/issues/39412

> Eventually, I hope we can remove macro_rules!. That will take a long time, and would require a 2.0 version of Rust to strictly adhere to our stability guarantees.

I don't think you're being charitable to the Rust team. They are quite vocal about how important they think API stability is. But they're also vocal about how important it is to get it right. For those reasons, they're upfront about what APIs are guaranteed to not change, and what APIs are still experimental. As an outsider, I think they do an excellent job of being transparent about the process.

Having used Rust off and on for the better part of 4 years through a lot of tumult and change, I agree with you.

In most cases the kinds of breaking changes and immaturities that the Rust core team communicates clearly to the ecosystem would be represented as nothing but silence and a giant dump of breaking changes suddenly in a lot of other ecosystems.

Sometimes the Rust team seems to pay a price for transparency, in that it gives people visibility, and thus something to make an opinion about and/or complain about. In an information vacuum there's nothing to opine about except the vacuum itself.

As annoyed as I often was, and occasionally still am, about different shapes of API stability in the language/libraries; I'm extremely happy the Rust team has maintained a commitment to developing everything out in the open.

To be clear, the new macros will be different than the old ones, and the two systems will coexist. People will be encouraged not to use the older macro style, but it's not going to be removed, so your code will not break.

There's no serious plans for a Rust 2.0, even though people bring it up from time to time.

> I won't want to abandon code just because v2.0 will get deprecated and shiny v3.0 will become the new standard next year. […] Forcing people to rewrite any code(esp. complex macros) will not gain Rust any goodwill longterm.

That's why nobody is planning to do so anytime soon. The quote you posted explicitly says that.

The author also created The Little Book of Rust Macros[1], which is an invaluable resource. They also are one of the top answerers for Rust questions on Stack Overflow[2]. Altogether a great asset for the community!

[1]: https://danielkeep.github.io/tlborm/book/

[2]: https://stackoverflow.com/users/42353/dk?tab=answers

The concept of a "token tree" as a separate stage between ordinary tokenization and full parsing seems interesting on its own: was it based on another system or invented for Rust? For example, you could imagine implementing incremental parsing by writing your parser as a transformation on a token tree.

Isn't it how Lisp macros work?

Yep. Although in some lisps you can write reader macros that control tokenization to add new syntax.

Yep! I have also been pointed at this PhD thesis: https://www.cs.utah.edu/~rafkind/papers/dissertation.pdf which defines a macro system on a uniform Algol-like syntax with intermediate "reading" and "enforestation" phases and corresponding syntax representations (the "read" representation being pretty much the same as the token tree representation here).

Cool! Any particular reason why you'd shuffle like that, as opposed to using a circular buffer? With IndexOffset, you've almost got a circular buffer going already.

I still find myself most comfortable with D's compile-time function evaluation (CTFE) and mixins. In D, one use of mixins is more or less an eval function: take any string and evaluate it as code at the invocation site.

http://dlang.org/mixin.html

It's nice that you need very little new syntax to do compile-time tricks in D. The same syntax that works at runtime also works at compile time. If mixins worked with the AST instead of strings, it would be almost the same as Lisp macros.

I'm also kind of happy to see that D seems to have inspired Rust macros a little.

Hi. I'm a lead developer on the Shaka Scheme project that hopes to accomplish implementing Scheme R7RS, which also has hygienic declarative macros.

R7RS Scheme has `syntax-rules` which is very similar in structure to this macro system. It's quite elegant as it is, in my opinion, but I'm also wondering how good the debugging facilities for it are. Are they good?

Scheme's syntax-rules was the direct inspiration for Rust's macro system, actually. As for debugging, I don't write my own Rust macros very often (and when I do, they're not very extensive), but I'd characterize the experience as "okay". Rust reports errors in macros as originating from the unexpanded line of code, which is useful for macro users but less so for macro authors. But you can ask the Rust compiler to emit the code that it sees post-expansion, if you want to inspect the generated output directly.

There's a `trace_macros` macro that you can use for debugging. It's somewhat useful.

The colored diagrams use to explain contexts and hygienic macros are a neat idea.

The intro to macros starts with a recursive macro. Not a good approach for an intro. Recursion in macros is usually more of a toy than a useful tool, anyway.

No, it starts with a non recursive macro that does less stuff, and builds up to the recursive one.

This resource is a "learning by example" resource, and it teaches the macro bit by bit starting from a simplified version and building up to the recursive and complex part, teaching concepts along the way. That's a great way of structuring it.

I actually think it's great that it starts by showing something that looks really impressive, it immediately made me think of list comprehensions in Haskell. I was very keen to keep reading as soon as I saw that.

Reply on news.ycombinator.com