Self Modifying Code as an alternative to macros | Hacker News Reader

Self Modifying Code as an alternative to macros

matklad.github.io

78 pointsmethyl31 comments

Click bait title!

The author knows full well, and even acknowledges in the very first sentence, that this article is not about self modifying code as commonly understood. This article is about generated source code.

Before CI became prevalent, I was against checked-in code generation and it took me a while to unprogram that bias and re-evaluate. Its been a big help in keeping runtimes fast with a python program at a previous job and at keeping build-times fast in an open source project of mine [0]

I wish there was a simple way in Rust to make proc macros optionally generate code that gets checked in, allowing for most of the macro costs to be shifted to dev-dependencies. While some macros might be generating code for very dynamic parts of applications, others like clap (CLI parser) are less likely to change and could benefit from it.

[0] https://epage.github.io/blog/2019/10/speeding-up-rust-builds...

> Alas, counting in macro by example is possible, but not trivial.

It's not stabilized yet, but RFC3086[1] introduces macro metavariable expressions, which make counting trivial!

The example macro becomes:

    macro_rules! define_error {
      ($($err:ident,)*) => {
        #[derive(Debug, Clone, Copy, PartialEq, Eq)]
        pub enum Error {
          $($err,)*
        }
    
        impl Error {
          pub fn from_code(code: u32) -> Option<Error> {
            match code {
              $( ${index()} => Some(Error::$err), )*
              _ => None,
            }
          }
        }
      };
    }

You can see this in practice on the playground[2].

[1]: https://github.com/rust-lang/rfcs/blob/master/text/3086-macr... [2]: https://play.rust-lang.org/?version=nightly&mode=debug&editi...

Thanks for the link, that seems very useful. I recently had to work around this by transforming a list (a, b, c, d) into something like 1+1+1+1+0 to get a count. What this RFC proposes looks so much less painful.

There's much cooler approach: https://veykril.github.io/tlborm/decl-macros/building-blocks....

  macro_rules! count {
      () => (0);
      ($odd:tt ($a:tt $b:tt)*) => ((count!($($a)*) << 1) | 1);
      (($a:tt $b:tt)*) => (count!($($a)*) << 1);
  }

It uses bit twiddling to avoid generating very big AST and eventually crashing the compiler. Though I have to admit using `${count()}` is better.

Generating code is a powerful tool that I think everyone should have in their toolbox — use it wisely of course, but don’t avoid it on principle.

This describes an interesting approach that I hadn’t seen. I personally wouldn’t have considered putting the result in the same file because I usually like to gitignore any and all generated code, mostly to reduce phantom merge conflicts. However, some colleagues have the opinion that those conflicts are trivially resolved by rebuilding and thus don’t matter, and having diffs of the generated code show up in pull requests adds value.

What do others think, check in generated code or put it in gitignore?

> However, some colleagues have the opinion that those conflicts are trivially resolved by rebuilding and thus don’t matter

This is how mistakes happen, and when dealing with things like serialisation is an absolute _minefield_

> nd having diffs of the generated code show up in pull requests adds value.

They do have a point here, but is there a reason that the generated code diff couldn't be done separately to being commited?

> What do others think, check in generated code or put it in gitignore?

Always ignored, no questions asked. Generated code is a build artifact, the same way an object file or a jar is. It just happens to be human readable.

Your build system should be 100% bulletproof and pick up changes to your input files and correctly regenerate the output files for you, anything less should be considered a p0 bugfix. If you need to cache these files for whatever reason, do so at a different level (sccache/container/whatever).

If you're going to check it in, at least have a CI job which rebuilds it from scratch, then fails if there are any changes.

I'm not persuaded of the value of having diffs in source control, but i haven't looked at concrete examples. This would be an interesting post for someone to write.

For performance, i'd rather follow a distributed caching approach. Hash the true source, look for /some/nfs/mount/or/s3/path/${hash}.tar.gz, if it exists, unpack it into the generated source directory, if not, generate the source, tarball it, and upload that to the said path. Takes some care to hash the right things (needs to include the version and configuration of the generation tool, etc). Again, you could have a CI job checking that the caches are correct if you're worried.

I like checking it in; on balance, I think it makes life nicer:

Simpler to get started running the code, less compute wasted running generators (this is especially nice when doing something like bisect), and it's nice to be able to just see all your code, right there in the repo.

I actually think the biggest downside is the diffs -- often there's no easy way (e.g. in GitHub) to hide all generated code, which can be annoying.

Merge conflicts don't bother me at all, since they're so easy to resolve; I agree with your colleagues there.

(Though that would be trickier if you had files mixing generated and hand-written code! Contra the original article, my instincts/advice would be to always keep generated code well-isolated from hand written code, ideally using some standard convention [e.g., all generated stuff is in /generated/ path].)

Keeping generated code in a segregated source code repository, with the advantage of inspecting diffs without file pollution, should be technically easy.

Commands to take snapshots of every build ("git add *", "git commit", "git rev-parse" on the actual source repository, "date" etc.) are easy to script.

Now that rust-analyzer has "expand macro recursively", you actually can see what a macro expands into fairly easily, so I don't think the unreadability of macros is much of a concern anymore these days. A bigger problem is the error messages you get when you write invalid code inside one.

Since the blog author is the primary developer of rust-analyzer, he must know about it. I think he might still have usability concerns about macros in rust IDEs.

I have been relying on cargo expand for macro debugging, let me check what utilities rust-analyzer has for this use case. Thanks for sharing.

EDIT: wow, it works exactly as you advertised, thanks a bunch again for sharing, such a nice productivity boost!

matklad's writings are such a gem!

I think this is a great investigation into a problem that's somewhat self-inflicted by rust, kind of like how so many of those GoF "design patterns" were simply a response to the lack of expressivity of Java.

For example, I think this problem doesn't even really make sense in Zig, where types are simply runtime values, and you have the full language available to you in a "compile time" pass. That approach is not without issues of its own (e.g. the article's point about IDE insight into the code, but writ even larger), but it's neat to see a different exploration in the design space, and how certain choices can make some issues just melt away.

Disclaimer: I'm both a rust and zig newb, and while I think I know roughly how I'd do this on the Zig side, I might be missing something.

I would probably manually update the enum and the corresponding match statement. If possible, I'd write a test to check that the two don't get out of sync.

I'm not too familiar with Rust so I'm not sure if the test code would be simpler than the macro solutions or the hack, but I think it would be.

Iterating over an enum requires a macro (strum::EnumIter is a popular one), which you’d need in order to ensure the test code automatically covers all cases without needing updates. This does only work when your variants are non-data variants or data variants whose type implements Default, but it does allow you to force yourself to stay in sync.

Edit: you might could also do some shenanigans using integers, `as` conversions, and #[repr(u8)], but it seems like it would get nasty and potentially unsafe

An unsafe block is significantly better than the code proposed in the article.

I know some people get upset when unsafe is used when there's a "safe" alternative, but unsafe isn't too bad in test code, is it?

The concerns of undefined behavior, data races, use-after-free, etc. seem a lot less consequential in test code.

Unsafe isn't too bad in any code. The point of unsafe is that it is a declaration that the programmer has verified the code is safe. It is certainly easier to verify something along the lines of:

    unsafe { transmute::<u32, Error>(error_code) }

than it is to verify that the weird macro code in the article is correct. There is no data races, use-after-free, etc. in that block, it does exactly what the article needs. To me, Rust programmers seem to have a weird notion that just the word "unsafe" is enough to conjure demons and monsters into otherwise totally normal code.

You'd think so, but that code is far into nasal demon territory. Both because the default byte representation of an enum isn't stable, and because an out-of-bounds enum value (wherever it comes from) is always UB.

An iterator macro seems fine? You could apply it only during the tests, for example, so it's not like it needs to be exposed in your library.

I think Walter got it spot on by not including any type of macro in the D language.

Personally waiting for his version of "Goto Statement Considered Harmful" but for the macros.

Just to be sure: goto is perfectly fine for error handling in C.

I think that mixins - both template mixins that essentially copy/paste declarations (including functions) and the mixin statement that takes a string to be compiled as source code - can be as weird to follow as macros in C.

Contrived example:

    import std.stdio;
    mixin template Func(string s, string r)
    {
        mixin("string " ~ s ~ "()" ~
        "{" ~
            "return \"" ~ r ~ "\";" ~
        "}");
    }
    mixin template Comb(string name, char c=',')
    {
        mixin(function string(){
            import std.conv;
            string r;
            enum counter(size_t x=[__traits(allMembers, mixin(__MODULE__))].length)=x;
            string suffix = to!string(counter!());
            r ~= "mixin Func!(\"Foo" ~ suffix ~ "\", \"Hello\");";
            r ~= "mixin Func!(\"Bar" ~ suffix ~ "\", \"World\");";
            r ~= "string " ~ name ~ "() {";
            r ~= "return Foo" ~ suffix ~ "() ~ \"" ~ c ~ " \" ~ Bar" ~ suffix ~ "();";
            r ~= "}";
            return r;
        }());
    }
    void main()
    {
        mixin Comb!("Mixer");
        writeln(Mixer());
    }

It is just that the language has more features than C so they aren't needed as often. But in my eyes at least the mixin statement is basically C macros on steroids :-P.

I am not familiar enough with Rust to understand what is going on here -- is this like Ruby's never-closed classes and metaprogramming?

No virus checker likes self modifying code

It’s not that simple. Also, the article isn’t about binaries at all, but about code generation at build time.

> virus checker

That industry is founded purely on fear and very little actual expertise. They should not fulfill an example role in your mind and nobody should actively listen to them when considering the future.

Is it just me that doesn't see a problem with the manual approach at all? Surely, there may be a better way to keep this in sync, but this is not very high effort.

Unrelated, but I can't help but think this should use try_from or even potentially panic. If you get an error type you don't recognize, that could pose a risk.

At least with the manual approach you can decide if particular ranges are safe to ignore or not and do something different. Granted, I'm dreaming up scenarios that are very likely outside the author's intention, so I don't think my thoughts on the matter necessarily refute anything being said.

The API in the article is try_from by another name... but leaving the question of panicking to the caller seems right to me. In the "I'm trying to respond to the error" case maybe unknown should be a panic, but what about when you share code with your log parser?

I think the difference is that the try_from implementation can’t be exhaustive because the input value space is much larger than the enum space, so nothing really requires you to add the new variant to that impl when you add the variant to the enum. This provides a non-macro solution that forces the two to stay in sync, which is pretty nice if you either don’t like macros or don’t feel comfortable building them yet. I agree with the author it’s also definitely more readable and obvious than a macro, even if the whole approach makes me feel a bit weird at first blush.

As you hint at in your last paragraph, it’s probably hard to come up with a blog-sized example for any programming technique. If 10-15 lines were really the extent our scope, we probably wouldn’t need anything more advanced than assembler. Some amount of imagination is necessary to consider whether an idea is useful in production code, so we all have different examples in mind.

Personally, I’ve had cases where three or four places had to be kept in sync, and the list was changing frequently enough that something like this became useful.

Even then, it’s still a judgement call whether code generation or (if available) macros are worth the complexity. The decision might also heavily depend on the experience of the team.

Reply on news.ycombinator.com