Lisp Meta-Programming for C++ Developers: First Macros | Hacker News Reader

Lisp Meta-Programming for C++ Developers: First Macros

deque.blog

84 pointsdeque-blog27 comments

Imagine the power of your whole programming language available to you at compile time. That is a Lisp macro. It gives you the ability to write simple and elegant code, which can then be compiled down into the unreadable nasty code after, while still being as fast as if you wrote the nasty code by hand. Another term for this is zero-cost abstraction. What is nice in Lisp is you can also use the (macroexpand ) function to see what Lisp code will actually be generated.. it's Lisp all the way down!

If you look at the Servant library [1] for ClojureScript, they wrapped the callback-hell mess that is Javascript web workers (read: the only way to actually multi-thread on the web), into beautiful Go-style channels, via the use of macros.

[1] https://github.com/MarcoPolo/servant

Typically you are limited at compile time because Lisps -- especially Clojure -- do not allow the same computing resources you get at runtime to be used at compile time, so there are some clear limits.

This can be true in C++ as well, but those limits are quite high and are easily set to something else. Perhaps Clojure also allows you to reset what these limits are.

Do you have any links documenting resource limits in Clojure macros?

I have plenty of personal experience trying to do things at compile time that are simply not available, and while I haven't looked up the details in a while, I do recall there being such a limit, but perhaps you have to make some settings on the compiler to get around. Macro expansion has relatively small limits in Clojure for things like memory usage.

The first part of this is not true.

You can execute arbitrary code at macro-expansion time.

You can also recompile while the process is running, which would include re-macroexpanding.

I think you misunderstood what I'm saying. Typically, the macroexpansion is a compile time process that uses different machine resources than the runtime processes of the same language.

Not in a lisp it isn't, compiling very often happens at runtime, in which case you're compiling in the same process as the code that is running at that time, in which case the same resources are available to both.

This is how you get a REPL and other forms of runtime evaluation.

This is kind of the nub of being a dynamic programming language, if it can't do this, it isn't a dynamic programming language. Lisps are generally dynamic programming languages.

Clojure-script might be an exception because of how it is implemented, but Clojure proper on the JVM certainly supports this type of behavior.

There must be a setting in regular Clojure on the JVM for controlling how much recursion or memory usage is available during macro expansion, because I've hit that limit very easily in the past. It's been about a year since I used Clojure much, but I doubt that's changed.

That makes more sense. It's possible that this is a design decision by Rich Hickey, I'm not aware of the intricacies of how the Clojure macro-expansion works. However, that sounds a lot like you hit a bug in the Clojure compiler (or in general needed to beef up the allowed memory of the JVM it was running on, compilation obviously isn't cheap... everything should all be in the same resident memory).

I would expect to be able to do (eval `(defun ,function_name () (complicated_macro ,@args))) or (compile `(defun ,function_name () (complicated_macro ,@args)))) at runtime in Common Lisp, without even really considering the consequences beyond whether it is necessary.

It's supposed to work a little bit like if you had an implementation of GCC in the same process as your C program, and you could call GCC from a function to compile some new C code for you to add features or replace existing code. It's powerful + dangerous.

In my case, it was definitely a problem of resource limits during compilation only, since the same logic would work fine if not a macro and run at runtime, so not a general case of the JVM not setup to handle the work, but specifically macros not able to handle the work.

C++ obviously has a limit on template depths as well, but they are so high that rarely does anyone get close to the defaults. Clojure probably has very limited defaults (assuming that defaults are the cause).

I can't make sense of what you are trying to say. In c++ you typically have no access whatsoever to the compiler at runtime. Through dynamic linking and some hacking about you can do this but it is all very non-standard.

In lisps you very typically have complete access to the compiler, though this is not universal. Resource limits on a running process are a separable issue.

We are talking about the resources of the machine and which of those resources are available during compile time compared to those which are available at runtime.

Yes, but you start off with an incorrect premise, and it's hard to understand what point you are trying to make.

In a very typical lisp system, for example, exactly the same resources would be available at runtime as your original compile (in fact, the boundary between these two is pretty fuzzy and often might be the same process anyway)

Likewise, C++ typically doesn't have any runtime access to compilers at all, so what resources are available to them or not is a bit moot.

If the point is that you might deploy on a different machine or configuration, well - that's true, but I can't see the relevance.

You might be pointing out some oddities of Clojure, I suppose, but that's not a great model for "typical lisp".

Indeed my experience is mostly with Clojure, which I mentioned because my parent commenter referred to ClojureScript. Glad to know a "normal" lisp handles this differently.

If you redefine define as a macro which evals the definition during macro expansion, you can have the full runtime at Compile-Time. State and IO included!

https://github.com/billsix/bug

http://www.lulu.com/shop/bill-six/computation-at-compile-tim...

Sometimes the situation is completely reversed: you get a lot more computing resources on the development system where the code is compiled, than on the target system where it ultimately runs (e.g. small, embedded target).

The computing resources required to process the code (macros, and whatever) are usually unrelated to the computing resources used by that code when it runs.

The computing resources required to process the code (macros, and whatever) is usually unrelated to the computing resources of the application.

A program that works with gigabytes or data might need only megabytes during its compilation.

(It's not easy to imagine practical circumstances where it would be reversed, with those exact numbers. Some algorithms you might want to use at compile time do explode in memory size, I suppose.)

That is getting quite normal nowadays; training a model in machine learning typically takes way more resources than running it.

However, if you need gigabyte to expand your macros, it is likely you need megaseconds, too, and then, it is wise to use 'precompiled headers', and save your 'preprocessor output'.

Defmacro is convenient. Debugging when the lack of hygiene leads to weird errors is not.

It happens once ever blue moon, but every time I would kill for a proper syntax-case for CL. Racket's is fine, but I'd be happy with a one like the plain R6RS as well

Side effects are no issue if every local variable name in a defmacro is generated with (gensym).

Exactly, you have to specify what you want. Evaluating an expression argument multiple times is probably a mistake, as is not having a new gensym for local variables ... until it is not, e.g. for anaphoric macros (binding some value to "it", https://en.wikipedia.org/wiki/Anaphoric_macro, you can still argue that that's bad style of course).

And if you're tired of doing that by hand, you can... write a macro that'll do that for you. In fact, those were already written, and are available in many libraries as well as featured in many books.

See e.g. Alexandria[0] for with-gensyms (automatically binding your variables to gensyms) and once-only (similar, but ensures the value will be evaluated only once).

Also note that lack of hygiene is actually a useful feature - it allows macros to capture variables from code passed to it as well as the environment they expand it. Some cool stuff can be done that way (some uncool foot-shooting too, though).

[0] - https://common-lisp.net/project/alexandria/draft/alexandria....

> Also note that lack of hygiene is actually a useful feature - it allows macros to capture variables from code passed to it as well as the environment they expand it. Some cool stuff can be done that way (some uncool foot-shooting too, though).

Doug Hoyte's excellent book _Let Over Lambda_ has a lot of examples.

Oh yes. This book is very fun, in the "what on Earth did he just do here" kind of way.

Syntax-case allows you to introduce bindings into the lexical context (breaking hygiene in a very controlled manner). It does not work with lists, but with syntax objects.

This is an age old (in lisp circles, anyway) argument that has never been resolved, and probably never will. Hygiene has pros and cons, after all.

I remember trying to explain the power of Lisp macros to a friend who at the time knew C++. I did not even know template-metaprogramming was possible, back then; that probably would have made it a lot easier.

Reply on news.ycombinator.com