How Lisp macros differ from static code-generation and metaprogramming
brandonbyars.com
brandonbyars.com
Huh? That's just blatantly wrong, and I (and many others) have used the C preprocessor to create multi-level macros in quite powerful ways. See for example http://blog.nelhage.com/2010/07/implementing-an-edsl-in-cpp/ or the insanity Linux's tracing macros have implemented: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6...
His article has a point, but he's not doing C macros justice -- they are quite powerful and can be used in really cool ways, even if people tend to look at you funny when you do :)
Think about it.
> they are quite powerful
For small values of "quite". C macros are strictly less (and by large margin powerful than Lisp macros. One can't even use #ifdef inside #define!
The C macro system was already too powerful, and one of the things that hurts languages is a preprocessor of virtually any kind. The badness of the C preprocessor starts much earlier with things like include files, and the fact that you can't statically analyze the code in any reasonable way.
The benefit from the preprocessor is absurdly small. With that said, I think the benefit of the Lisp preprocessor is pretty small too. C would be harmed more by its omission due its compilation model, but as we've seen with languanges in its family, like C#, its unnecessary.
In a decade or less, the Lispers will see this light too.
Lisp doesn't have a preprocessor, Lisp is a preprocessor. You can program with it, but thinking about it as a programming language (ASCII or Unicode Strings to be Interpreted) is at least partially wrong.
"In a decade or less, the Lispers will see this light too."
It hasn't happened in the last half century, so how much do you want to bet on it?
This would be news to McCarthy. McCarthy on several occassions, including the History of Lisp, refers to Lisp as a programming language. I've never heard him say Lisp is a preprocesser. I certainly am no expert compared to McCarthy. Maybe you are?
It hasn't happened in the last half century, so how much do you want to bet on it?
It's importance has ebbed and flowed over time. This is natural of language features. It was a decade ago where all the rage was template metapgrogramming in C++. Now, outside of a few library writers, most people don't touch it. This isn't to say it won't be popular again some time in the future.
I should probably ammend my statement in that light and say that we'll likely see a major ebb in the coming years.
Are you talking about the importance of Lisp macros here? That's what you appear to be doing. I don't see a decline in enthusiasm for use of Lisp macros among Lisp programmers.
"It's importance has ebbed and flowed over time. This is natural of language features. It was a decade ago where all the rage was template metapgrogramming in C++. Now, outside of a few library writers, most people don't touch it."
The reason that few touch template metaprogramming in C++ is because it is a painful, limited, and in pretty much every way sucky experience. Many of the simplest tasks you wish to perform are either nearly impossible, or often drastically increase the amount of code you have to write. Many of us tried (present company included) and moved on because we figured out what a waste of time it was.
Lisp macros, on the other hand, are a very powerful tool that can reduce the amount of code you have to write (and provide new abstractions to do useful things with, and probably at least a dozen other things I haven't mentioned). They are not perfect, and they can be abused, but they shouldn't even be mentioned with C++ templates in the same breath (unless you are talking about how much better Lisp macros are :-D ).
I see your ammended statement/prediction and raise you with this one: I predict that Lisp macros will continue to be at least as important as they have ever been (at least to Lisp programmers) in the next ten years.
The reason you can't statically analyze C code that uses macros has to do with the preprocessor's broken design - it is a separate stage from the compiler, and there's no way for the two to communicate.
All this actually means that Lisp macros can be used for static analysis (via macroexpand), in a user-extensible way, without needing to add extensions to the compiler or stages to the compilation process.
I don't understand why you cite C# as a good example of something that doesn't have code preprocessors. Visual Studio has T4 (Text Template Transformation Toolkit), which is a dumb joke.
But back to the point, I partially agree with what you say about C, but I think you misunderstand where the crux of the problem. It's not that they're separate stages -- I believe they are in most Lisp implementations (but I may be wrong), but rather that you can't do macro expansion w/o the context of the full program, i.e., within the context of the build system -- yet the effect is global.
I don't think you know anything about the use-cases for Lisp macros. It's a standard practice to use them to generate documents (most CL HTML generating libraries are just sets of macros), configuration information (SQL schemas, XML) for other systems, and programs in other languages (for example, this is how Parenscript compiles Lisp to JavaScript).
"rather that you can't do macro expansion w/o the context of the full program, i.e., within the context of the build system -- yet the effect is global."
I don't understand this.
Maybe I don't know the Lisp uses cases for writing conceptual docs, but I must say that this sounds horrible. But clearly each to their own... when all you have is a hammer. I'd much rather use just about any templating system (and there are many) than any language (or preprocessing system) that I've ever encountered.
I don't understand this.
In C, the only way to know the impact of macros on code is to build it (preprocess it). And macros have global effect on the code, so you don't know, when looking at any piece of code in C, what it actually does, unless you look at it preprocessed -- for the most part. This means pulling in the build system, since obviously you don't know what is being pulled from where otherwise.
It's hard to quantify "general good" of something complex as programming language macro system with all consequences included.
But it's quite easy to see which one is more powerful wrt expressiveness.
Your primitive macro system _forces_ me to write code like this:
#ifdef CONFIG_AUDITSYSCALL #define INIT_IDS \ .loginuid = -1, \ .sessionid = -1, #else #define INIT_IDS #endif
#define INIT_TASK(tsk) \ { INIT_IDS }
instead of more natural.
#define INIT_TASK(tsk) \ {\ #ifdef CONFIG_AUDITSYSCALL .loginuid = -1, \ .sessionid = -1,\ #endif }
> The badness of the C preprocessor starts much earlier with things like include files, and the fact that you can't statically analyze the code in any reasonable way.
Static analysis hardness or easyness has _nothing_ to do with preprocessing, because static analyzer can preprocess code in a very same way compiler will so both will analyze the very same stream of tokens.
> The benefit from the preprocessor is absurdly small.
This is subjective and unquantifiable.
In a decade or less, the Lispers will see this light too.
For example:
(defun sum (x y)
`(,x + ,y = ,(+ x y)))
(sum 4 5) ==> (4 + 5 = 9)
If you want to confirm this, you can change the + and = to any other symbol and it would still display. E.g:
(defun sum (x y)
`(,x frob ,y != ,(+ x y)))
(sum 4 5) ==> (4 FROB 5 != 9) (mac until (test . body)
`(while (no ,test) ,@body))
I think combining backquote-comma's s-exp-making and s-exp-destructuring powers is a viable way to abstract s-exps from their representations as linked lists/arrays/whatever sequence you want to use.
http://www.lisperati.com/looking.html
"Flip, flop, flip, flop..."
The same can be said for classic Forth. Indeed, the enthusiastic way that Lispers describe macros I find eerily similar to how I used to hear Forthians describe their language. Except in Forth the relevant term isn't macro but "defining word". Indeed, I think programmers in both camps see the proper use of macros/defining words as a sign that one has passed the beginner stage of using the language.
A classic compiled Forth word (ie, function) would consist of the word's name followed a string of addresses to execute; how those addresses were laid down, where they pointed, etc. were totally open to programmer control -- both during compilation and afterwards. It's been a couple of decades since I looked at Forth code, but I don't think there's anything that could be done in Lisp macros that couldn't also be done easily in Forth as well.
I think the big difference between the two -- at least, so far as the macro topic goes -- is that Lisp starts with an assumption that code and data can and should be interchangeable while Forth was initially developed for the machine control (embedded systems before they were called embedded systems) and thus code was merely a vehicle for creating applications.
Forth is an interpreter that can be switched into compilation mode, Lisp is a compiler that can be switched into interpreter mode. [edit: thinking of that, this more reflects how I have used the two than what they are. Lisps start in an eval loop, too, yet they feel different to me]
Forth is postfix, Lisp is prefix.
Forth programmers know their stacks are the environment, but rarely think of it as an environment. Lisp programmers know their environment is in some stack-like data structures, but rarely think of those stacks.
Both are quite flexible.
Forth often runs on the bare metal, and exposes everything below. Lispers are customarily more shielded.
Both encourage code to be treated as data and vice versa (without even looking at macros), but Lisp builds on the lambda calculus to do that, while Forth just sees code and data as being contents of memory.
In Forth if there's a bunch of data flowing into a word, you might put the number of arguments on the top of the stack and the other arguments below. Lispers would run away screaming from this manual and error-prone approach.
Lispers like to build e.g. their their own object system, if necessary. And in Forth it's easy to tag on a garbage collector---from within the language.
I hope that's some meat to back up my assertion. I hope I got some good examples. What I really want to say, is that the languages feel related in spirit. But less like brothers, more like to generals from opposing armies still respecting each other.
Interestingly, both languages wound up with a schism of framework enthusiasts (CLOS & "big Forths") vs. minimalists (Scheme & Moore loyalists (for want of a better term)).
As an aside, your bunch of data example would run afoul of Forth catechism. The normal response would be to use a dictionary address or even address and offset to access a data structure. Antagonism to deep use of the data stack is so prevalent that Moore eventually reached the point with his chip designs of implementing the top of stack as a couple of on-chip registers -- if what you want isn't within the top couple of stack entries, so the logic goes, then you've failed to factor your design properly. Perhaps this preoccupation with very low level operations is an aspect where Forth programming might be considered the opposite of normal Lisp programming.
And you are right about my example with the deep stack. I probably did not remember correctly. The variable array was probably something with passing around an address plus length on the stack---while Lispers prefer that as one object.
1. Macros aren't unique to lisp. They're most developed and used in the lisp family, but you don't need to have a homoiconic language to have macros. The Mirah programming language--statically typed Ruby on the JVM--has non-hygenic macros, I know there's been a number of efforts to add macros to coffeescript (so far without success), and there are a number of academic languages that also have them. None are in widespread use, but they exist.
2. Metaprogramming's power is primarily limited by the restrictions of the host language. The best example I know of is Io where pretty much anything other than commas and parentheses is up for grabs. Example: http://anttih.com/blog/2010/10/29/json-in-io.html
3. User definable mixfix operators are in the sameish domain. Unfortunately, they're almost always mentioned in passing by people who know what they are so I don't have a good reference for introducing what they are and don't have direct experience working with them. I'm just mentioning them as something to be aware of if you're interested in general metaprogramming concepts. Here's the closest thing to an intro I could google up: http://maude.cs.uiuc.edu/maude1/manual/maude-manual-html/mau...
2. Io is great but the depth of it's runtime meta-programming facilities make it horrifically inefficient.
As for Io's efficiency, I think that's more about the implementation than in the language. Javascript (prototypal inheritance) and Smalltalk (message passing) aren't that far off and both have decent performance. I know Steve Dekorte was working on getting Io running in javascript in December but I don't think it's a high priority project for him.
Perl6 also has macros (http://en.wikipedia.org/wiki/Perl_6#Macros) though they haven't been implemented in Rakudo yet. Also Ioke states that it has macros.
2. Metaprogramming's power ... The best example I know of is Io where pretty much anything other than commas and parentheses...
Touched on Io introspection/metaprogramming before on HN:
ref:
* http://en.wikipedia.org/wiki/Perl_6_rules
For (3) look at how Haskell defines new operators. I find Haskell's use of completely new operator-squiggles plus type classes for something faintly similar to C++-style overloading, much more sensible than say, C++-style overloading.
> Except, of course, that doesn’t work. The preprocessor
> only makes one pass through the file, meaning a macro
> can’t call another macro.
macro.c:
#include <stdio.h>
#define LOOP(n) for (int i = 0; i < n; i++)
#define PRINT10 LOOP(10) { printf("%d", i); }
int main(void) { PRINT10; printf("\n"); }
gcc -Wall -std=gnu99 macro.c && a.out
0123456789
Ignoring his poorly protected macros, and taking nothing away from Lisp, am I missing some subtle point he's making, or is the author just flat out wrong here? I think he's just wrong, and this makes it hard for me to even read the rest of his argument.
(a nonsensical example, as I can't think of a proper one right now)
#define PRINTFOO PRINTBAR
#define PRINTBAR PRINTFOO
PRINTFOO
The ANSI C standard says a macro, while it is being expanded by the preprocessor, ``is painted blue'' (is no subject to further expansion).
I'm not sure, but probably it follows the C macros are not Turing-complete.
EDIT: on the other hand, a LISP macro system is a Turing-complete programming language that's geared towards manipulating lists of symbols and values (which, incidentally, is LISP program).
Perhaps I was wrong; perhaps the phrase was in K&R C? Can't find the book online @_@;;
#include <stdio.h>
#define LOOP(n) \ int i; \ for (i = 0; i < n; i++)
#define LOOP10 \ LOOP(10) printf("%d\n", i)
int main(int argc, char *argv[]) { LOOP10; }
Here it is:
(defmacro for-duration ((seconds) &body body)
`(handler-case
(bt:with-timeout (,seconds)
,@body)
(bt:timeout () nil)))
Use as:
(for-duration (10)
(loop
(print "Infinite loop! where is your Godel now?")))
(defun for-duration (seconds body)
(handler-case
(bt:with-timeout seconds
(body))
(bt:timeout () nil)))
(for-duration 10
(lambda ()
(loop
(print "Infinite loop! where is your Godel now?")))) #define for_duration(seconds, body) \
{ \
pthread_t tid_task, tid_watcher; \
\
void* task(void* arg) { \
body ; \
pthread_cancel(tid_watcher); \
return NULL; \
} \
\
void* watcher(void* arg) { \
sleep((seconds)); \
pthread_cancel(tid_task); \
return NULL; \
} \
\
pthread_create(&tid_task, NULL, &task, NULL); \
pthread_create(&tid_watcher, NULL, &watcher, NULL); \
pthread_join(tid_task, NULL); \
}
for_duration(10, {
while (true) {
printf("Infinite loop! where is your Godel now?\n");
}
});
for_duration(10, while (true) printf("something\n") );
int i = 0;
for_duration(10, {
while (true) {
printf("%d\n", i++);
}
});
void with_timeout(unsigned int seconds, void (*func)(void));
#define for_duration(seconds, body) { \
void func() { body ; } \
with_timeout((seconds), &func); \
}
(defmacro execute-in-reverse (&body body)
`(progn ,@(reverse body)))
(execute-in-reverse (print "Hi") (print "Middle") (print "Bye"))
"Bye"
"Middle"
"Hi"
But my example is trivial, too. I could go on to swap individual parts of my forms around, remap them to other forms depending on various conditions, etc.
In addition, people often talk about Lisp macros, but they neglect to mention the power of reader macros, which allow you to go beyond Lisp's basic AST look-and-feel.
No, of course. C macros only see strings; you need to parse a syntax tree to do your reverse example (like lisp macros).
No argument from me.
>Lisp's macro system allows you to create new syntax, including changes in flow control.
Well, C macros seem to be able to manipulate control flow. You can't break up an expression to do that (not smart enough to parse), but you can rearrange expressions that are given whole.
Just for shit & giggles I'd thought I give it a go in Io (http://www.iolanguage.com):
executeInReverse := method (
m := call argAt(0)
stmts := list() // list of statements (ie. messages)
loop (
rest := m next // rest of messages after current
stmts append(m setNext) // get current message
m := rest
if (m == nil, break) // exhausted statements when "nil"
)
stmts reverseForeach (n, doMessage(n))
)
executeInReverse( writeln("Hi") writeln("Middle") writeln("Bye") )
If you've ever built a string in T-SQL or PL/SQL and then exec'd it, you're probably much closer to being productive with macros than you think.
Of course, this is probably because dynamically generating SQL is something that I've done. Other programmers probably have their own mental models that warp into lisp.
Now there's an idea...
I can see why they are as powerful as claimed.
(loop for y = 0 then z
for x from 1 to 5
sum 1 into z
finally (return (values y z)))
(loop for x from 1 to 5
for y = 0 then z
sum 1 into z
finally (return (values y z)))I agree with the definition at http://en.wikipedia.org/wiki/Metaprogramming
A key tradeoff is that they must be expanded at run-time, and in doing so they incur a performance penalty that doesn't exist with compile-time macros.
However, we have no useful English operators to take the quoted words and change them into something else meaningfully. Though a non-English set can be argued to exist: http://creativeservices.iu.edu/resources/guide/marks.shtml If those were English themselves that would arguably complete the loop, but I would consider that stretching the point; these were explicitly invented to avoid the use of English and also translate to any other character-based language.
X-bar theory says that in all natural languages, Noun Phrases, Verb Phrases, and depending on who you read other types of phrases all have the same form; a single Head (which may be composite) and a Tail with zero or more elements. Never mind that they've just discovered S-expressions. The point is that in the natural language program:
multiply the radius squared and pi
But seriously, take a look at Quines and Hofstadter's Godel, Escher, Bach
(let ((let '`(let ((let ',let))
,let)))
`(let ((let ',let))
,let))
i wish authors didnt craft their arguments around C/Java.