Ask HN: What metaprogrammable language do you/would like to use? | Hacker News Reader

Ask HN: What metaprogrammable language do you/would like to use?

44 pointskarmakaze73 comments

Like many folks here, I probably spend too much time reading about, trying and contemplate using many new languages. The (overlapping) paradigms that interest me are metaprogramming, functional, and (newer) statically-typed. I decided to focus only on metaprogrammable ones from now on (as a time saver and to step up).

I made a list and ordered them by how much I would be interested in using them (which combines my curiosity with current/expected adoption).

My short-list for metaprogrammable ones are:

  Clojure
  Elixir
  Nim
  Crystal
  Rust (would be higher if I did more low-level work)
  Pony

I left most other functional ones off my list because that's an exploration in itself for another time. I was surprised that I put Clojure and Elixir first given my preference for static types. Of all the kitchen-sink features that Nim has, I can't accept camelCased == under_scored names otherwise it could have been first. Ruby is notably absent as I use it and am looking for something better/different.

For future adoption, I think interoperability is a key factor, whether it's with C or in a VM runtime (e.g. JVM, CLR, BEAM, v8).

Which metaprogrammable language do you use or are most interested in using? How compact are your programs (i.e. how extensive do you metaprogram)?

View on HN

I just add "Julia" for this list;

Ideal for Scientific computing. ( LLVM based; Optionally typed; Dynamic )

https://julialang.org/

https://docs.julialang.org/en/v1/index.html

Julia Metaprogramming:

https://docs.julialang.org/en/v1/manual/metaprogramming/inde...

Julia: "Building a Language and Compiler for Machine Learning" ( compiling to GPU; TPU )

https://julialang.org/blog/2018/12/ml-language-compiler

"Why Does Julia Work So Well?"

https://ucidatascienceinitiative.github.io/IntroToJulia/Html... ( "Core Idea: Multiple Dispatch + Type Stability => Speed + Readability" )

I'd like to say I'm a huge fan of Julia, and don't have much experience with metaprogramming in other languages.

I'm sure I do more of this than necessary, but much of my code ends up being "generated functions".

@generated foo(x, y) ... end

When you write an `@generated` function, you write code that creates a Julia expression. Using "x" and "y" in the function give you the types of these arguments. Defining parametric structs lets you pass any information you want (eg, array sizes) to specialize code and algorithms.

A very useful library: https://github.com/MikeInnes/MacroTools.jl

Mike Innes produces so many cool Julia libraries!

Flux is easily the coolest ML framework I’ve used, and I’ve just discovered Lazy.jl.

I came here to say Julia as well!

I didn’t really understand or get the point of metaprogramming, but the more Julia I’ve been writing the more I’ve seen fantastic uses of it and the more inspired I get.

Plus the language itself is a real pleasure to write.

I had no idea that Julia was good for metaprogramming. I'm so glad I asked. Thanks for the links-added to my reading list.

Python has way more metaprogramming capabilities than people think.

Of course you can intercept pretty much anything with __dunder__methods (attribute missing, method access, instantiation, etc), you have full instrospection of pretty much anything (functions, objects, modules, call stacks...) and you also have monkey patching, decorators, metaclasses, bytecode injection...

But while we are far away from Lisp based languages, import hooks give you access to the ast of any module, and allow to use your own parser, before the result is loaded, to decide what to return. Which means you can pretty much make your own syntax and import it like a regular module.

However, the reason those tools are not very well known is that the community regards magic as dangerous as it is powerful, and everybody agrees on using it sparingly.

Hence the only popular libs that use a lot of magic are ORM (lot of metaclasses, dunders, etc) and test libs (monkey patching, bytecode injection, ast parsing), but not a lot more. I don't know of any popular lib that actually uses the import hooks to create a DSL.

The Hy language (HyLang) is a good example of this. A lisp syntax to transformed to an AST that is executed as Python is. Hy is an interesting project and a good bet if you really want to use Keras and TensorFlow in a Lisp language.

I use Red (full disclosure, I'm on the team), and have used Rebol (Red's direct ancestor) since 2001. When I find something that works, I stick with it, though I also keep an eye out for new things.

Red's heritage comes from Lisp, Forth, and Logo. It's homoiconic and metacircular (it is its own meta language). Where Rebol was strictly interpreted, Red can also be compiled, and has hygienic macros. But you don't really need them. They're nice for moving things to compile time, but Red puts a twist on Lisp's sexpr model that obviates the "need" for them in most cases. Technically, you can say Red uses an fexpr model, but a more human-friendly way to say it is that "Everything is data until it is evaluated.", and you have a lot of control over when evaluation occurs.

For interop, Red has a system-level dialect called Red/System. It's a C level language and is a dialect of Red. That is, Red is high level and is used to define the Red/System dialect, which is used to implement Red. It's the circle of life. :^) You can also compile Red as a library and call into it via an API, so you can use it as an embedded language, or doing things like the Excel example in this blog entry about Red's macros: https://www.red-lang.org/2017/03/062-libred-and-macros.html which also mentions one of the easiest ways to preprocess input, with the `system/lexer/pre-load` feature.

ASTs are mentioned in a few comments, so I should add that while you can certainly do that with Red, it's another thing that can often be avoided entirely. With other langs and tools, you almost have to take that approach, to make it manageable. With Red, there is a function called `parse` which consumes input and supports BNF-like rules to process it. `Parse` is a Red dialect, and the rules are just data it interprets. Rather than building ASTs (though there are some cool examples, and tree-rewriting systems out there), just interpret the input directly.

If it sounds like I'm against metaprogramming, while being on a team creating a language that supports it deeply, that's not the case. Metaprogramming is a great tool for thinking, but it can also make things much harder to debug and maintain. For real work, use what makes your intent clear, and avoids as much complexity as possible.

Personally, I use Nim to develop domain specific language for VM, JIT, emulators and neural networks.

I'm currently writing a compiler for deep learning with both AOT (emitting Nim code) and hopefully later JIT (emitting LLVM IR) capabilities complete with SIMD support. I don't see how I could do that in another language.

My biggest successes:

- include a quite maintainable JIT for x86_64 (compared to asmjit and xbyak I don't need to parse or codegen the C++ code): https://github.com/numforge/laser/blob/master/laser/photon_j...

- a DSL for neural network: https://github.com/mratsim/Arraymancer#sequence-classificati...

- a matrix multiplication BLAS written from scratch competitive with OpenBLAS and MKL on select matrix shape (2000x2000) but that can also support int8/int16/int32/int64 and not just float32/float64 thanks to metaprogramming. Expanding to new SIMD architecture (ARM) is very easy:

--> benchmark: https://github.com/numforge/laser/blob/master/benchmarks/gem...

--> Metaprogramming AVX512 support: https://github.com/numforge/laser/blob/master/laser/primitiv...

The most important thing for me for metaprogramming is being able to operate on the AST directly

I use Common Lisp for its really insane metaprogrammability (reader macros, compiler macros, ordinary macros, and the MOP to edit the object system) and the ability to bend to the problem that I am trying to solve. It has decent interoperability via its CFFI interface.

> Of all the kitchen-sink features that Nim has, I can't accept camelCased == under_scored names otherwise it could have been first.

Just to clarify for the general public (because this is often a source of confusion and misunderstandings): first letters are case-sensitive in Nim, so you can do `var car: Car` (where `car` is a name of the variable, and `Car` is a type).

When it comes to style-insensitivity, I also thought this would be a problem when I discovered Nim, but now—2 years later—there was not even one situation where that would cause a problem. (And if you use different styles of the same name to mean different things in other languages: this will bite you sooner or later)

Nim's standard library is written uniformly in camelCase style, which is what I also accepted for my code (even though I come from snake_cased Python).

So why this even exists? To make easier to use libraries written (in another language, e.g. C) in another style, while keeping the uniform style in your code. Basically, this feature makes it easier to use just one style consistently.

To conclude:

If this is the only (or just a major) thing that keeps you from using Nim: I would suggest you to reconsider and give it a chance.

Absolutely! I hear this complaint a lot for people who seem to read about the language without ever trying it out. It does sound a bit crazy to begin with, but not having to write code that's a mish-mash of styles while still being able to use a library written by someone who didn't adhere to the style guide is a really nice feature.

I don't see how this is required for one drop and if that's the use case, it sho_uLd only be used there and not everywhere. It makes it harder for tooling and for searching for all references. Inconvenient, like spaces in filenames.

One of the main benefits is in wrapper libraries for C. Another one is using libraries that have a naming style that I don't like.

Searching for names is not a problem as people don't mix different styles within the same project. Also there's nimgrep for this.

Finally, getting a compiler error when I try to use variables called should_run and shouldrun in the same scope is a feature, not a bug: it encourage using less misleading names.

If it treated 'same scope' to mean something more like a module (i.e. library), that would be better.

Also, it should never consider hell_owl and hello_w_l to be the same, although each could match hellOwl and helloWL. Basically it takes the style-insensitivity too far.

And yet, after using Nim for many years hell_owl == hello_w_l has never been a problem. Again, it warned me in few occasions when I tried using confusing names like new_dir and newdir.

There are style guides and nimpretty to help further.

If you are asking about learning metaprogramming as a paradigm I would rather suggest Racket. Lisps are famous for their metaprogramming capabilities and you have Clojure at the top of your list but writing macros are way easier in Racket than in Clojure. This is primarily because Racket IDE provides good debugging and tracing tools. Clojure is notorious for its cryptic error messages. This certainly doesn't help while you try to do code transformations. Once you learn the fundamentals you can apply them in (somewhat) more mainstream functional programming languages like Clojure or Rust.

But Racket doesn't have macros a la Common Lisp, which are the most powerful way to achieve metaprogramming in the Lisp family.

Nonsense. Racket has hygienic macros just like CL.

CL doesn't have hygienic macros.

I stand corrected. Racket does.

Racket has syntax-case[1], which is hygienic, but allows you to break hygiene if you need to.

Racket also supports defmacro[2] like Common Lisp, and it's actually implemented using syntax-case.

I would argue that syntax-case is at least as powerful as defmacro.

[1] https://docs.racket-lang.org/guide/syntax-case.html

[2] https://docs.racket-lang.org/compatibility/defmacro.html

Interoperability + meta-programming? You have to go with the king, Common Lisp! There's a bunch of implementations for different platforms [0]. It's also much simpler to write macros in a homoiconic language like a lisp, than using something like Rust's elaborate macro system.

Although your phrasing ('expected/future adoption') makes it sound like you value picking a language that will have loads of jobs available. Clojure has a few solid niches (same number of users as Kotlin IIRC) and is 'symbiotic' with JVM and JS environments. Rust is growing and has supposedly 'safe' BEAM NIF support. Elixir is popular with the Ruby crowd. Although I have to say, 'meta-programming' & 'functional programming' isn't exactly the hottest job market filter.

[0] https://common-lisp.net/implementations

I would recommend that you definitely look into Scala. The language that was fundamentally designed to be "scalable" with and/for frameworks. Check this resource for more information.

- https://scalameta.org/

- https://speakerdeck.com/itakeshi/metaprogramming-in-scala-th...

- https://geirsson.com/post/2016/02/scalameta/

- https://github.com/milessabin/shapeless

- https://vimeo.com/217863345

If you're interested in functional programming and meta-programming as paradigms, then you should really focus on functional first. The reason is that functional is all about transformation of data structures as a whole (as opposed to altering them bit by bit as in procedural programming) and meta-programming is about treating programs as data and transforming them. You really can only do advanced meta-programming using functional techniques. It's no accident that Lisp is the granddaddy of both paradigms.

As for which language to use to explore these paradigms, the answer would have to be a modern Lisp, i.e. Clojure or Racket. Of those two Clojure is more about being a "pure functional" language and Racket more about meta-programming... in fact Racket has been called a "meta-language".

> You really can only do advanced meta-programming using functional techniques.

My `loop for (symbol value) on bindings by #'cddr` and "push onto list in a loop, nreverse and return at the end" macros beg to disagree.

Metaprogramming is just like any other programming, except that the output is fed to the compiler. The way Lisp is compiled, you can even use code generated by code in code that generates other code.

That said, functional programming does indeed fit well with most of the stuff you do when writing code that writes code.

The generation of globally fresh variables (the Lisp community uses the famous

   gensym

function for this purpose) is notoriously non-functional. Template Haskell uses a variant of the IO monad to hide the global state involved in the definition of gensym.

Haxe has pretty good metaprogramming capabilities. It has expression macros (compile-time evaluated calls that return expressions to be inserted at call place) as well as type-building macros (build fields and/or define whole new types). Both have access to compiler and system API and information about the context (e.g. expected type).

Oh, and regarding "how compact are your programs", well every time I see boilerplate code or reflection usage that cannot be refactored into something nice without sacrificing clarity and/or performance, I use meta-programming to generate it instead.

Usual suspect here is any kind of "support" code, like serialization, RPC, dependency injection and such.

I have used Nim extensively and done a lot of metaprogramming in it. The biggest risk to metaprogramming is that it allows crazy syntax that you have to learn separately. Nim does very well here to enforce limitations which work well in reducing this risk.

Comparing Nim's metaprogramming to Rust's shows what I mean. In Rust you can write macros that act on token streams, so as long as the token stream is valid it can be used. This allows libraries like the typed-html package[1].

In Nim you cannot do this unless you put the HTML in a string literal. This limitation created a syntax that is far more idiomatic, admittedly at the expense of familiarity for users of HTML:

    buildHTML():
      html():
        head():
          title: text "My webpage"
      body:
        p(class="red"): text "Hello World"

In my experience this works quite well, you can see a larger example in our Forum's code which is written using a SPA framework called Karax[2].

Please don't let style insensitivity discourage you from using this wonderful language.

1 - https://docs.rs/typed-html/0.2.0/typed_html/

2 - https://github.com/nim-lang/nimforum/blob/master/src/fronten...

Please don't let adherence to stlye-insensitivity limit language adoption.

If it was up to me I would have removed this feature long, but not because it's a problem, only because of people like you who judge programming languages based on what they see at the surface.

There are still plenty of people who dislike Python because "significant whitespace, bleh" with no objective reason why they dislike it except "it gets messed up when I copy the code". That's a tooling problem, just like with style insensitivity, it's not hard to search in a style insensitive manner and in fact all tools should allow this mode for convenience.

Okay, so what's the regexp for finding all the style insensitive versions of myVariable? Something like

  m_?[Yy]_?[Vv]_?[Aa]_?[Rr]_?[Ii]_?[Aa]_?[Bb]_?[Ll]_?[Ee]

I never want to type a regexp like that just to find all occurrences of a simple var name. Even if a plugin did it for me, it would still be slower and it doesn't help me on the command line.

This list is missing Haxe, which is strictly-typed (with type inferencing), supports object-oriented, generic, and functional programming and is highly extensible thanks to meta-programming (macros). Interoperability is where it excels, since it compiles to to multiple language targets/platforms, including VM bytecode.

Note: Haxe does not typecheck macros.

See https://haxe.org/manual/macro-ExprOf.html

For example, this seemingly ill-typed definition (which should return Int as suggested by the type paramater, but in fact returns a string) successfully typechecks:

  class StringGetter {
    public static macro function getString():haxe.macro.Expr.ExprOf<Int> {
      var s:String = "";
      return macro $v{s};
    }
  }

That's not entirely true. `ExprOf<T>` itself is not checked in either the macro function argument or return type, but that's only meant for static extension (and maybe some other features?).

Expressions generated by your macros will be parsed by the type checked. You won't be able to do

    var s:Int = StringGetter.getString();

Because you'll get a compilation error there (http://try-haxe.mrcdk.com/#48EDb).

> Haxe does not typecheck macros.

That is a weird statement. Haxe expression macros by definition return untyped AST to be processed further normally by the typer, just like hand-written code. And it will be type-checked. And then there's an AST node existing exactly for type-checking (or, rather, type unification) that you can generate to insert custom checks: https://haxe.org/manual/expression-type-check.html

Incorrect. To quote the page you've linked: "For the most part, this type is identical to Expr, but it allows constraining the type of accepted expressions." or otherwise put: ExprOf only affects macro arguments, not return value. What you want is this:

  class StringGetter {
    public static macro function getString() {
      var s:String = "";
      return macro ($v{s} : Int);
    }
  }

Considering only production code here.

Do use: Common Lisp, Racket, Python, Ruby

Would like to use: Julia, Rust, Elixir

I metaprogram when I see I have already built a mini-language/pattern in my code. Even when writing Lisp, I begin with simple code without extending the language, and only when I have written a bit do I consider looking for patterns.

Out of curiosity, why do you have Python on your list? Python doesn’t really support metaprogramming? I’ve written my fair share of Python but don’t remember ever coming across anything meta in the language. Even if it has some passing implementation, it definitely doesn’t have it in the way Lisp/Julia/Rust/etc have it.

> it definitely doesn’t have it in the way Lisp/Julia/Rust/etc have it.

Yes, but then again, Rust, Julia, and Lisp are _high_ standards of metaprogramming. Python's approach to metaprogramming, like most of its other parts, is simple and concise. For simple tasks that are repeated often, Python sometimes provides easy ways to abstract the how and show only the what. While Python does have metaclasses — which, unfortunately or otherwise, I've never needed to write — I've sometimes found myself reaching for simpler tools like decorators, magic dunder methods, even iterators and generators. And, of course, there's always the ability to generate Python AST, or have something generate it for you.

Examples:

1. the new-in-3.7 feature: dataclasses[0];

2. the library that inspired it: attrs[1];

3. the (de)serialisation library with a strikingly simple design: marshmallow[2];

4. the leaky-abstraction of SQL that makes it somewhat composable: sqlalchemy[3];

5. Google's python bindings for Protocol Buffers[4]; and, of course

6. the lisp-in-Python: Hy[5];

----

[0]: https://www.python.org/dev/peps/pep-0557/

[1]: http://www.attrs.org/

[2]: https://marshmallow.readthedocs.io/en/3.0/_modules/marshmall...

[3]: https://docs.sqlalchemy.org/en/13/orm/extensions/declarative...

[4]: https://developers.google.com/protocol-buffers/docs/pythontu...

[5]: http://docs.hylang.org/en/stable/language/internals.html

You can do some nice things in Lua [1], but the one I'm most interested in at the moment is probably Terra [2]. Its concept is that it's a language like C, but uses Lua as a sort of preprocessor Language, so you can use complex macros, templates, etc. There's even a java-like class-system, if I remember correctly.

[1] https://github.com/darkwiiplayer/moonxml

[2] http://terralang.org/

It depends a bit what you mean, but Groovy is quite interesting from a meta-programming point of view. Much of the syntax can be completely reprogrammed at run time through an object's 'metaClass', and you can write AST transformations that deeply rearrange the code at compile time.

Although it is certainly not as pure as the Lisp-like languages, it can be an incredibly fast and easy way to create a DSL to solve your problems in a very idiomatic way.

I never thought of Groovy for metaprogramming but it's used to make DSLs so it makes sense. My only run in has been with Gradle which is so slow and/or a memory hog so didn't leave a good impression on me.

It's unfortunate that most people meet Groovy through Gradle. It's a really confusing and unpleasant experience and I think is responsible for a lot of the bad rap that Groovy gets. It's actually way better than Gradle makes it look!

When I hear metaprogramming my brain just goes "Lisp". So I am a bit surprised to not see CL in your list. :)

> Ruby is notably absent as I use it and am looking for something better/different

what issues do you have with ruby? it's the de-facto standard for meta-programming

I often use things like `define_method`, `constants`, `method_missing`, etc in select places.

last fun thing I did was adding `let` and `it` to rails's tests.

Every large, long -lived Ruby app I've worked on eventually needs more performance, better concurrency, and would benefit from static typing. It's great that each company got to the point of 'a problem you want to have'. I'm just looking for the best of both worlds: good to start, good to scale.

Nim. I use it a lot and the case insensitivity has never been a problem and it's often useful.

It's worth to mention Factor programming language - https://factorcode.org/

https://en.wikipedia.org/wiki/Factor_(programming_language)

https://github.com/factor/factor/

http://andreaferretti.github.io/factor-tutorial/ - Good introduction and Metaprogramming section.

Haskell with TemplateHaskell is definitely worth giving a shot.

You can find good examples of its use by searching for reverse dependencies of template-haskell package[0].

One of these is my tiny experiment that features program synthesis by given type[1]. Recently I added an ability to handle sum types and product types (a.k.a. Either and tuples) -- see 'rewrite' branch.

[0] https://packdeps.haskellers.com/reverse/template-haskell

[1] https://github.com/8084/haskell-holes-th

Template Haskell is nice, but doesn't allow higher meta-programming, eg. meta-meta-programmming.

Definitely take a look at Forth: each word in the language can have separate compile-time and run-time behavior, and the compiler is simple enough that it's easy to extend it with your own compile-time words. https://www.forth.com/starting-forth/11-forth-compiler-defin... is a good overview.

I would love to try Rebol (or Red) for some meaningful task.

I am surprised nobody has mentioned C++

I know its hated passionately, but I learned about metaprogramming from it.

Also, imo, Racket is the best language for metaprogramming.

OCaml has some metaprogramming facilities through the ppx system. It wouldn't be my first choice for metaprogramming, but it's there if you just want to know the full map of how different languages do it. I have found the ppx system a little bit underdocumented and challenging to use, but it can be very powerful. OCaml fits all of your other requirements.

I've run into OCaml before but haven't taken the time to get into it. If I were to, I think I might look at F# but I have no idea if the metaprogramming is the same.

I like Elixir overall as a language and love it as a platform. That said, the metaprogramming feels passable for me. I would much prefer to write and read things in a Lisp but it works fine in Elixir

I'm interested in trying Rascal, but it's meta-meta: https://www.rascal-mpl.org/.

I am surprised Prolog did not make the list. I have used SWI Prolog on a profession project in the past.

Its quite powerful for certain types of problem domains like expert systems.

Elixir all the way. :)

Using github and gradle plugins I metaprogramm java. Can't recommend it through. It's extremely error prone and hard to reason about.

Scala has the most advanced meta-programming facilities. And Scala's MP is widely used.

If you are interested in research-prototype languages, https://convergepl.org/ is interesting. But Converge is no longer under active development.

Which features make it the most advanced?

Compile-time MP and run-time MP. Support for AST and quasi-quote MP, and support for LMS (= lightweight modular staging).

You haven't seen any Lisp yet, have you?

Compile and run-time MP and quasiquoting are basic features; I'd argue that a language without these doesn't even qualify as "supporting metaprogramming".

Sure, I'm intimately familiar with the Lisp family of languages. It lacks features that I prefer in a modern meta-programming language, in particular an expressive typing system with type inference, and a platform / eco-system like the JVM.

I stand corrected, and I agree with you here on both accounts - I'd love if CL in particular had a better type system standardized (current one sits in a kind of uncanny valley) and a better ecosystem. CL implementations are solid, but the mindshare unfortunately isn't there anymore.

Cool, I wasn't aware of Scala's LMS. Thanks for sharing!

JavaScript. It is awesome.

Not sure if it's sarcasm. True, JavaScript is fun to meta-program. But it's hell to debug.

You never know about sarcasm. But I just wanted to give a shoutout to good old JavaScript as it lends itself beautifully to metaprogramming and it is actually widely used unlike some of the more academic languages mentioned here.

Something better than ruby? No such thing ;-)

Reply on news.ycombinator.com