Hacker News Reader

Comment by pakkarde | Hacker News Reader

After having written a somewhat complete C parser library I don't really get the big deal about needing meta programming in the language itself. If I want to generate structs, serialization, properties, instrumentation, etc, I just write a regular C program that processes some source files and output source files and run that first in by build script.

How do you people debug and test these meta programs? Mine are just regular C programs that uses the exact same debuggers and tools as anything else.

View on HN

>I don't really get the big deal about needing meta programming in the language itself. If I want to generate structs, serialization, properties, instrumentation, etc, I just write a regular C program that processes some source files and output source files and run that first in by build script.

This describes exactly what people don't want to do.

But exactly why?

Because after enough hands have touched a codegen script, debugging it becomes impossible.

In jai you use the same language for programming and metaprogramming. The compiler knows how to execute the bytecode it generates. The compiler also has a builtin debugger for the bytecode.

If you just walked up to me out of the blue and asked "what computer language do you know is the worst for processing strings?", well, technically I might answer "assembler", but if you excluded that, my next answer would be C.

Furthermore, you want some sort of AST representation, at one level of convenience or another (I include this compgen-style "being 'in' the AST" to be included in that, even if it doesn't necessarily directly manipulate AST nodes), and C isn't particularly great at manipulating those, either, in a lot of different ways.

A consequence of C being the definitive language that pretty much every other language has had to react to, one way or another through however many layers of indirection, for the past 40+ years, is that pretty much every language created since then is better than C at these things. C's pretty long in the tooth now, even with the various polishings it has received over the years.

C# (strictly, Roslyn/dotnet) provides this in a pretty nice way: because the compiler is itself written in the language, you can just drop in plugins which have (readonly!) access to the AST and emit C# source.

Debugging .. well, you have to do a bit more work to set up a nice test framework, but you can then run the compiler with your plugin from inside your standard unit test framework, inside the interactive debugger.

Yes, this is the same approach Ryan Fleury and others advocate, and it's perfectly good:

> Arbitrary compile-time execution in C:

> cl /nologo /Zi metaprogram.c && metaprogram.exe

> cl /nologo /Zi program.c

> Compile-time code runs at native speed, can be debugged, and is completely procedural & arbitrary

> You do not need your compiler to execute code for you

https://x.com/ryanjfleury/status/1875824288487571873

MS DOS choice of / for commandline arguments and \ for paths always hurts my eyes

The only benefit that some (certainly more rare) compilers can provide is type metadata/compile-time reflection. Otherwise, totally.

This gets unwieldy quick as soon as your compiler host and target are different platforms.

I don't know about zig bit the power of lisp is that youre manipulating the s-expressions or to put it another way, you're manipulating the ast. To do that in C you'd need to write a full C parser for your C program that processes source files.

I used to do that in Python with the numba jit. Write Python code that generates a Python code that then gets compiled.

It's a fragile horrible mess, and the need to do this was a major reason for me to switch away from Python. It's a bit like asking why we don't just pass all arguments to functions as strings. Yeah, people write stringly typed code, but it should rarely be necessary, and your language should provide means to avoid it.

Well put. I always have the feeling that any language which has an `eval` function or an invokable compiler can do meta program. That said, I think the "big deal" is in UX/DX. It's really nice to have meta programming support built-in to the language when you need it.

Whether you consider it a big deal or not is up to you, but with zig's approach you don't have to write/maintain a separate parser, nor worry about whether it's complete enough to process your source files.

I don't know a lot about debugging zig comptime, though. I use printf-style debugging and the built-in unit test blocks. That's all I've needed so far. (Perhaps that's all there is.)

> How do you people debug and test these meta programs?

I couldn't find any other answer than using @compileLog to print-debug [1]. In lisp, apparently some implementations allow to trace macros [2]. Couln'd find anything about Nim's macro debugging capabilities.

This whole thing looks like a severe limitation that is not balanced by the benefit of having all code in the same place. Do you know other languages that provide sensible meta-programming facilities?

[1] https://www.reddit.com/r/Zig/comments/jkol30/is_there_a_way_... [2] https://stackoverflow.com/questions/44872280/macros-and-how-...

In lisp, macros are just ordinary functions whose input and output is an AST. So you can debug them as you would any other function, by tracing, print debugging, unit tests or even stepping through them in a debugger.

To debug macros in Nim, you'll likely need to print arguments and expansions at compile-time, inspect the output, change things to see what happens, repeat...

https://nim-lang.org/docs/macros.html#toStrLit%2CNimNode

https://nim-lang.org/docs/macros.html#astGenRepr%2CNimNode

https://nim-lang.org/docs/macros.html#dumpAstGen.m%2Cuntyped

https://nim-lang.org/docs/macros.html#treeRepr%2CNimNode

https://nim-lang.org/docs/macros.html#dumpTree.m%2Cuntyped

https://orlybooks.com/books/changing-stuff-and-seeing

> I just write a regular C program that processes some source files and output source files and run that first in by build script.

Cool, you now invented your own DSL and half-baked meta programming macro language for something that shall have been in the language to begin with.

In addition, any complex interaction between your "own made template engine" and the native code is now a pile of hack. E.g write a generic function: Good luck to interpret any error based on the typing.

Code generation is almost consistently the worst solution to a meta-programming problem.

> Cool, you now invented your own DSL and half-baked meta programming macro language

I'm not sure I'm following your statement. What he said was to use a C program to parse C code and emit additional C code. There is no mention of DSL.

> for something that shall have been in the language to begin with.

The whole point of this discussion is to debate on that.

I have no strong opinion (yet) but the meta program looks easy to understand compared to the pandora box of metaprogrammaing withing the language (since it requires standardization, limitations etc.).

I think the parsed c program is the DSL. You have to write a parser and compiler for the c-like source to the actual c code.

> What he said was to use a C program to parse C code and emit additional C code. There is no mention of DSL

Because it is always the same story:

- You start by writing a little meta-compiler to solve one specific codegen problem in a specific portion your code.

- Then you realize their is many slight variations of this problem in other areas of your project or other projects... because it is exactly what *Genericity* is all about. And we know that since literally the 1970s and freaking LISP.

- To avoid your meta-compiler to become a Frankenstein of options with endless hardcoded logic: you make it interpret some annotations in C comments, some preprocessors or some template files somewhere.

-> Congratulations: you invented your own half baked DSL.

I have seen that many time, in many places. Again and again. Often because there is a category of C programmers that would prefer to swim in their own shit instead of using few C++ templates.

Codegen is consistently a terrible solution to a well studied problem: meta-programming. The fear of the feature creep (templates, macros) shall never be a justification to create some half baked complexity monster that will alaways finish worst than the problem they try to avoid.

If you doubt about that: Just use a lexer or parser generator. Or better, the quintessence of codegen: Autotools. They are a perfect illustration of how terrible and how fucking unmaintainable Codegen is.

I can think of a single use case where a meta-compiler and a DSL are appropriate solutions: *Serialization* (Protobuf, Thrift, CapNProto, ...). Because in this precise case, you actually do want a language neutral way to express your interface: you want an IDL.

Currently here, Zig does the right thing: Comptime execution for meta-programming is one order of magnitude better than anything available in C or C++ before C++20

The notion of "something that shall have been in the language" is not properly defined, as you can basically start from that and arrive at C++. So it is perfectly fine to assume that compile-time programming does not belong into the language and write your own processor.

As for DSL, any DSL is a separate language that needs to be learned, so it is not very different from creating your own processor.

Because it goes hand-in-hand with your code self-describing with static reflection.

Reply on news.ycombinator.com