The existence of template metaprogramming is an argument for macros, not against them. It shows the lengths to which programmers will abuse and misuse a feature so that they can reach their goals.
When I was studying this issue, I came across femtolisp, which is a standalone Lisp by the author of Julia, Jeff Bezanson. Julia has metaprogramming so I wanted to look at it.
In this video [1] Professor Steve G. Johnson explains metaprogramming (around 10 min mark) and says "most of the time, don't do metaprogramming." He points out in the base Julia there are 36,798 methods, but only 138 macros and 14 generated functions (.04% of the code), so I have the impression it is still frowned upon.
This is an incorrect assumption. In compiled Lisps[0], macros are a compile-time construct, which manipulate the data structures that represent your code[1]. All manipulation occurs at compile-time. In interpreted Lisps, macro evaluation is temporally intertwined with program execution, but the two phases are logically distinct.
[0]: Compilation time in a Lisp is not necessarily similar to compilation in other languages. Working at a REPL does not imply that interpretation is happening. Clojure is an example of a Lisp that is exclusively compiled. Even when sending individual expressions to the REPL, they are compiled before being evaluated.
[1]: From a reply elsewhere in this thread: https://youtu.be/P76Vbsk_3J0?t=1333 See this video for an overview of the core of Clojure. The link starts with an overview of Clojure's data types and data structures (skipping the intro for the reasons behind Clojure). This leads directly into the evaluation model, which is based on the data structures of Clojure. Watch through ~1h:30m to get a feel for what macros are used for in Clojure's core. This gives a very good overview of the evaluation model for macros; this is generally applicable to Lisp macros outside of Clojure as well, though there are nuances to different dialects. I recommend that you start at the linked time, but if you want to jump straight to the evaluation model, you can jump to ~47m.
If I understand that correctly I don't get it, because I would still have to write the macro, which in C# I would write as a function that I call in my source code, which doesn't grow the syntax of C#, but grows the program to a DSL that solves my problem, albeit not as nicely as in a REPL.
Maybe this is all about the REPL and I don't use them or see any appeal in them. I like to write my text, look at it, and hit run. You write a REPL line and it disappears.
I'll keep learning. I'm sure the light bulb will go off eventually.
A Macro just lets you write a function that will be executed by compiler to manipulate such tree, meaning that you can have equivalent of adding syntax like "with/using" from some languages by writing a function that takes as arguments the object and a block of code, then re-emits the same block of code but adding object.Dispose() call at the end. Or whatever else you need.
But they are still normal functions - it's the time of execution that changes.
I think I'll try to get a new understanding of Lisp macros by trying to come up with a language feature in C# or Dart that I really want but can't implement without a lot of non-intuitive difficulty, but could easily in a Lisp style macro that the compiler could use to build in a new feature at compile time.
Clojure's only built-in conditional operator is 'if'. Despite this, we have nice short-circuiting or, and, when, unless, and other conditional operations in Clojure, defined as macros.
Clojure (and C# and most languages) eagerly evaluate their function arguments. You cannot write a function that short circuits. Let's consider the following Clojure:
(when false (infinite-loop))
But, when is a macro, which means that during compilation time, the compiler defers to the when macro. The when macro is near-trivial.
(defmacro when
"Evaluates test. If logical true, evaluates body in an implicit do."
{:added "1.0"}
[test & body]
(list 'if test (cons 'do body)))
So, when receives a list, '(false (infinite-loop)). when returns a list to the compiler, '(if false (do (infinite-loop)). The compiler sees that there is no more macro expansion to be done, so it emits the appropriate code to execute that operation. When we get to run-time, the if happily short-circuits and this ends up being a no-op.
This sort of conditional macro tends to be trivial in implementation, but highlights the difference between an eagerly evaluated argument to a function and a syntactic form passed to a macro for transformation at compile time. And if Clojure lacked a conditional construct, you could happily add it.
Continuing with the comparison between Clojure and C#. C# gained async as a new keyword which required compiler modifications to implement. Clojure got core.async as a library. Anyone could have implemented core.async on their own. Now, Clojure's core.async is CSP-style (similar to Go) and C#'s rewrites your code into a state machine on your behalf, which basically pretties up callback handlers for you. If you prefer that C# async style to CSP, you can introduce that construct in Clojure yourself with something that looks as "native" as core.async. If you want to use CSP in C#, you cannot create any syntax for this and so will never be able to make something feel native as async does.
Or another example would be something like C#'s using construct for IDisposables. If you wanted to implement using in a C# without it, you couldn't. You cannot create new control flow constructs. Something similar with Java AutoClosables is implemented as a macro in Clojure as well (again, if it didn't exist in the language, you can add it just like below). You can implement arbitrary control flow in Lisp macros in a way that would require syntax and compiler modifications in other languages.
(defmacro with-open
"bindings => [name init ...]
Evaluates body in a try expression with names bound to the values
of the inits, and a finally clause that calls (.close name) on each
name in reverse order."
{:added "1.0"}
[bindings & body]
(assert-args
(vector? bindings) "a vector for its binding"
(even? (count bindings)) "an even number of forms in binding vector")
(cond
(= (count bindings) 0) `(do ~@body)
(symbol? (bindings 0)) `(let ~(subvec bindings 0 2)
(try
(with-open ~(subvec bindings 2) ~@body)
(finally
(. ~(bindings 0) close))))
:else (throw (IllegalArgumentException.
"with-open only allows Symbols in bindings"))))
By dropping laziness in strict functional languages such as Scheme, SML, OCaml, Scala, and F#, the purity and semantic beauty of true functional programming is lost. When programming in Haskell, laziness is one of those things that you rely on all the time without noticing it; it is something you only realize once you miss it.
As for async/await, Erik led the effort to put it in C# and then Dart. I'm assuming he wanted to bring Haskell Continuation Monads [2] to the popular languages.
I'm building a hobby website, so I got into Blazor because I like C# and was interested in doing it client side. I gave up on it, not because of C# or bad implementation (it's actually great) but because Microsoft has a noisy and bureaucratic way of configuring code that drives me nuts. I went back to Dart because it is amazingly productive on the client side, but now this video [3] shared in this thread makes me think I should give ClojureScript a try.
[1] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.72....
[2] https://www.haskellforall.com/2012/12/the-continuation-monad...
Two parts here. First, and briefly a note on short circuiting and lazy evaluation. Second, and longer, a further discussion of Lisp macros.
First. Short circuiting is a special case of lazy evaluation, but is the most common exposure to lazy evaluation among programmers in the C family of languages. It is the property of conditional and boolean operations that evaluate only the minimum necessary to return a result. (or true IGNORED) only needs to evaluate the first operand to or, true, because the result of the or is true regardless of the value of IGNORED. This is common to all mainstream strictly evaluated languages I am aware of. https://en.wikipedia.org/wiki/Short-circuit_evaluation
Macros allow you to implement lazy evaluation in an otherwise strict language, but this is not a comprehensive explanation of their utility.
Second. I've watched pretty much all of Rich Hickey's presentations a half dozen times or more. Somewhere in the mix of more practice, another time through SICP, and watching Rich Hickey I started getting what homoiconicity means and the value prop of Lisp.
It's simultaneously very prosaic and very deep. Macros are pretty simple:
1. Access to the AST of a chunk of code for arbitrary processing.
2. At compile-time.
This is not exclusive to Lisps. I think pretty much any mature language offers a library or a built-in mechanism for getting at the AST of arbitrary code in that language. The "at compile-time" part is not necessarily built in to other languages. You absolutely can have a build system that has two (or more) passes, the first building a macro expansion framework and library of macros and the second processing the rest of your source code and rewriting the AST of the code in those files to do macro expansion.
Where Lisps are different:
1. The AST of the language is represented directly in the core data structures of the language. An operation in Lisp[0] is simply a list whose first element is an operator (one of a special operator, a macro, or a function in most Lisps) and whose remaining elements are operands to that operator. This makes AST manipulation easier, because you use the same functions/abstractions/interfaces as in normal programming, because you're directly using familiar data structures.
2. Separation of reader and evaluator (interpreter or compiler or either/both). The reader transforms textual representations into in-memory representations of values and data structures. The evaluator only ever receives these in-memory representations. Thus, it's easier to do programmatic manipulation of an AST. You never have to do code-gen of emitting source code glyphs to feed into a compiler that expects text input. Once past the reader, everything is an in-memory representation, so we never have to get back to text. That said, it's trivial to get back to text if you want, because Lisps also include a printer which takes an in-memory representation and prints the glyphs that represent their readable (i.e., can be read by the reader) textual representation.
3. The evaluation model explicitly supports tagging some code as a macro to go through macro expansion before being evaluated "normally".
4. It is idiomatic to use macros to solve code re-use problems that cannot be readily handled by standard function calls.[1]
These four Lisp traits are not exclusive to Lisp, necessarily. As I mentioned above, you could build a macro expansion and code generation library that is part of a build process for any compiled language. Eval, in interpreted languages, is as expressive as Lisp macros, but is incredibly poorly supported and unergonomic in comparison to Lisp macros.
It's an ergonomic and feasibility thing more than a capability thing. That said, POSIX shell is turing complete, so you don't need anything like C# or Lisp or Haskell or anything else we've discussed to get your computation done. The reaction you might have to comparing shell scripting to Haskell is not dissimilar to the reaction an experienced Lisp programmer might have to comparing AST manipulation and code gen through a library to Lisp's macro facilities.
[0]: Clojure, specifically abstracts the idea of callable, and there are more callable things than special operators, macros, and functions.
[1]: Languages such as Haskell and OCaml continue to push the boundaries of what can be handled by "standard function calls" (which term I hesitate to use, given its imprecision). That said, given that macros allow the use of a turing-complete language to generate inputs to the evaluator, they allow you to implement any language construct that is computable without modifying the evaluation infrastructure.
That's what drives me on. That and Rich Hickey's enthusiasm and the sense that he knows something really valuable that I don't. I watch a lot of videos as well, so I can't remember who said that programming languages were user interfaces between human and machine and are an attempt to talk to machines to get them to do things. In that sense, programming languages are a lot deeper than syntax convenience or efficiency. A good language allows us to talk to machine at a much higher level, and I get from functional programmers that this is what they find so attractive.
I will contact you if I get really stuck and when I finally get it just to share it with someone. Thanks!
edit: I've heard a lot about SICP but never looked into it. I think it's time, now that I have time
Videos: https://ocw.mit.edu/courses/electrical-engineering-and-compu...
Text: https://github.com/sarabander/sicp (links to epub and html versions at the top of the README)
I don't have a computer science degree and I think that has held me back some, but I try to overcome it by always trying to learn and never thinking I should already know something or feeling like I'm an expert. I don't have the patience at my age to go back to school and get a degree. That you attempted SICP a few times is a good sign that you don't give up and speaks to your success in finally understanding.
Compared to Elixir (another language I like) - it is homoiconic but it doesn't have this feature. I can't, for example, take an arbitrary function definition and update or define it in the REPL - because all functions must be defined in a module. However, I like these modules because I think it helps enable discovery of functions - it's much easier to find all operations you can do on lists or maps in Elixir than Clojure.
Also, I wouldn't necessarily call macros more powerful than functions. They're just different, and they enable different things. You cannot, for example, pass a macro around like you can a function. Personally I've never written them, but I use them every day because the core library defines macros. I view them as ways to let people smarter than me to give me tools I can use to write my programs.
As far as driving you on, I don't think Clojure is the end-all be-all language. I happen to like it. I also like Elixir. I know people who really love other, more standard languages though. I don't think you need to force yourself to love or "get" Clojure, but playing around with it can expand your mind a bit, especially if you've never used an FP before.
But if you want an FP that (IMHO) would be easier to transition to, Erlang was the first FP that I "got", afterwhich I tried Clojure again. The syntax is a lot more like traditional imperative programs which can ease the transition. Nowadays I would recommend Elixir over Erlang for several reasons: more modern with modern tooling, less syntax quirks, more active online community, lots of active development.
For me, the main benefits of Clojure are (other FP languages have some of these too):
1. Sharable, immutable state as the default.
2. A set of higher order functions that can do all sorts of data transformation.
3. Being able to use the same language on the web in client + server (without having to use Javascript).
4. Clojure gives me the ability to, more than other languages I've used, target the level of abstraction that I need for the problem at hand, which removes a lot of cruft.
5. Lack of special syntax: helps enable REPL driven development, makes it easier to seamlessly adopt new features from other languages.
6. It's hosted on two ecosystems I'm very familiar with - Java and Javascript.
As someone who has given similar advice when teaching metaprogramming in the past, I'll add some context to that: most people new to metaprogramming seem to reach for it all the time for some reason, in situations where the usual programming constructs are perfectly adequate. Imagine if any time you introduce people to multithreading, they start writing all their code as parallel threaded code (which happens to some degree, but is limited by factors include what a PITA it is). That's the context in which such warnings are deemed necessary.
Just like multithreading, manual memory management, and other great-power-great-responsibility programming concepts, macros shouldn't be the first tool you reach for, but when you need them you really need them, and they can make your life so much easier.
What you describe is actually used in a lot of modern software, usually self-modifying code refers to code that modifies itself while running, typically in a loop, not just generating fresh code :)
That's why I stopped doing it, but I totally get that in low level programming that it comes in very handy in some circumstances.
At the same time, when you really need a macro, you really, really need a macro because it can do things a function cannot like controlling when or if its arguments are evaluated.
There are multiple reasons to use a macro rather than a function. I won't describe them here* but suffice it to say that the statement "macros are no longer useful in modern programming" is completely false.
*See pg's _On Lisp_ for an in-depth discussion of macros.