Truly asking for help: can you help explain what can I do with macros that I cannot do with functions? Or, maybe, cannot do with high quality or low complexity using functions?
Spend 5 minutes playing with -> [0]. Appreciate how pretty it makes the code and how nice it is for laying out sequential operations. What a fun operator. Then try to implement it yourself as a function.
When you discover that you can't, you will have discovered the unique power of macros (note the docs link to source code so you can see the readable 8-line macro implementation). You've also learned a very practical skill, which is how to write threading macros that handle your special edge case fail conditions. It comes in handy from time to time.
[0] https://clojure.github.io/clojure/clojure.core-api.html#cloj...
But I don't think this is a bad example at all, I tend to think of macros as "extending the language/compiler", since they have access to the abstract syntax tree of their inputs, which is what is being done here.
It doesn't matter what specific combination of language features are supported. Macros mean you can implement things anyway. It doesn't really matter what the evaluation logic of the language is, a macro can create syntax that does what you want it to.
If there is a macro system available, a new operator can be implemented in any language. If there is no macro system, it depends what features the language supports.
It's possible to write pretty complex things like control flow abstractions using just structures and functions/methods, e.g. a CL-esque condition system in Java which only uses classes, static methods, and Java lambdas[0] for its syntax. Possible, but also IMO ugly when compared to the CL counterpart, because the low-level but irrelevant details (such as instantiation via `new` or generics) are still presented to the programmer.
[0] https://coalton-lang.github.io/20211010-introducing-coalton/
One fairly simple example is Clojure's `when`, which is an expression that starts off looking like
(when (= 0 1) (+ 1 2))
(if (= 0 1) (do (+ 1 2)))
It's instructive to see how much of e.g., Clojure is implemented as macros. https://github.com/clojure/clojure/blob/master/src/clj/cloju... shows pretty clearly the correspondence between the original form and the rewritten form in the definition of `when`. `defn`, which is so foundational to idiomatic Clojure that it's easy to mistake it for a built-in, is a much more complicated example.
[1]
Some other languages support similar, if not more powerful, compile-time facilities (such as Lisp macros), but those are outside the scope of this article.
When I was studying this issue, I came across femtolisp, which is a standalone Lisp by the author of Julia, Jeff Bezanson. Julia has metaprogramming so I wanted to look at it.
In this video [1] Professor Steve G. Johnson explains metaprogramming (around 10 min mark) and says "most of the time, don't do metaprogramming." He points out in the base Julia there are 36,798 methods, but only 138 macros and 14 generated functions (.04% of the code), so I have the impression it is still frowned upon.
This is an incorrect assumption. In compiled Lisps[0], macros are a compile-time construct, which manipulate the data structures that represent your code[1]. All manipulation occurs at compile-time. In interpreted Lisps, macro evaluation is temporally intertwined with program execution, but the two phases are logically distinct.
[0]: Compilation time in a Lisp is not necessarily similar to compilation in other languages. Working at a REPL does not imply that interpretation is happening. Clojure is an example of a Lisp that is exclusively compiled. Even when sending individual expressions to the REPL, they are compiled before being evaluated.
[1]: From a reply elsewhere in this thread: https://youtu.be/P76Vbsk_3J0?t=1333 See this video for an overview of the core of Clojure. The link starts with an overview of Clojure's data types and data structures (skipping the intro for the reasons behind Clojure). This leads directly into the evaluation model, which is based on the data structures of Clojure. Watch through ~1h:30m to get a feel for what macros are used for in Clojure's core. This gives a very good overview of the evaluation model for macros; this is generally applicable to Lisp macros outside of Clojure as well, though there are nuances to different dialects. I recommend that you start at the linked time, but if you want to jump straight to the evaluation model, you can jump to ~47m.
If I understand that correctly I don't get it, because I would still have to write the macro, which in C# I would write as a function that I call in my source code, which doesn't grow the syntax of C#, but grows the program to a DSL that solves my problem, albeit not as nicely as in a REPL.
Maybe this is all about the REPL and I don't use them or see any appeal in them. I like to write my text, look at it, and hit run. You write a REPL line and it disappears.
I'll keep learning. I'm sure the light bulb will go off eventually.
Clojure's only built-in conditional operator is 'if'. Despite this, we have nice short-circuiting or, and, when, unless, and other conditional operations in Clojure, defined as macros.
Clojure (and C# and most languages) eagerly evaluate their function arguments. You cannot write a function that short circuits. Let's consider the following Clojure:
(when false (infinite-loop))
But, when is a macro, which means that during compilation time, the compiler defers to the when macro. The when macro is near-trivial.
(defmacro when
"Evaluates test. If logical true, evaluates body in an implicit do."
{:added "1.0"}
[test & body]
(list 'if test (cons 'do body)))
So, when receives a list, '(false (infinite-loop)). when returns a list to the compiler, '(if false (do (infinite-loop)). The compiler sees that there is no more macro expansion to be done, so it emits the appropriate code to execute that operation. When we get to run-time, the if happily short-circuits and this ends up being a no-op.
This sort of conditional macro tends to be trivial in implementation, but highlights the difference between an eagerly evaluated argument to a function and a syntactic form passed to a macro for transformation at compile time. And if Clojure lacked a conditional construct, you could happily add it.
Continuing with the comparison between Clojure and C#. C# gained async as a new keyword which required compiler modifications to implement. Clojure got core.async as a library. Anyone could have implemented core.async on their own. Now, Clojure's core.async is CSP-style (similar to Go) and C#'s rewrites your code into a state machine on your behalf, which basically pretties up callback handlers for you. If you prefer that C# async style to CSP, you can introduce that construct in Clojure yourself with something that looks as "native" as core.async. If you want to use CSP in C#, you cannot create any syntax for this and so will never be able to make something feel native as async does.
Or another example would be something like C#'s using construct for IDisposables. If you wanted to implement using in a C# without it, you couldn't. You cannot create new control flow constructs. Something similar with Java AutoClosables is implemented as a macro in Clojure as well (again, if it didn't exist in the language, you can add it just like below). You can implement arbitrary control flow in Lisp macros in a way that would require syntax and compiler modifications in other languages.
(defmacro with-open
"bindings => [name init ...]
Evaluates body in a try expression with names bound to the values
of the inits, and a finally clause that calls (.close name) on each
name in reverse order."
{:added "1.0"}
[bindings & body]
(assert-args
(vector? bindings) "a vector for its binding"
(even? (count bindings)) "an even number of forms in binding vector")
(cond
(= (count bindings) 0) `(do ~@body)
(symbol? (bindings 0)) `(let ~(subvec bindings 0 2)
(try
(with-open ~(subvec bindings 2) ~@body)
(finally
(. ~(bindings 0) close))))
:else (throw (IllegalArgumentException.
"with-open only allows Symbols in bindings"))))
By dropping laziness in strict functional languages such as Scheme, SML, OCaml, Scala, and F#, the purity and semantic beauty of true functional programming is lost. When programming in Haskell, laziness is one of those things that you rely on all the time without noticing it; it is something you only realize once you miss it.
As for async/await, Erik led the effort to put it in C# and then Dart. I'm assuming he wanted to bring Haskell Continuation Monads [2] to the popular languages.
I'm building a hobby website, so I got into Blazor because I like C# and was interested in doing it client side. I gave up on it, not because of C# or bad implementation (it's actually great) but because Microsoft has a noisy and bureaucratic way of configuring code that drives me nuts. I went back to Dart because it is amazingly productive on the client side, but now this video [3] shared in this thread makes me think I should give ClojureScript a try.
[1] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.72....
[2] https://www.haskellforall.com/2012/12/the-continuation-monad...
Two parts here. First, and briefly a note on short circuiting and lazy evaluation. Second, and longer, a further discussion of Lisp macros.
First. Short circuiting is a special case of lazy evaluation, but is the most common exposure to lazy evaluation among programmers in the C family of languages. It is the property of conditional and boolean operations that evaluate only the minimum necessary to return a result. (or true IGNORED) only needs to evaluate the first operand to or, true, because the result of the or is true regardless of the value of IGNORED. This is common to all mainstream strictly evaluated languages I am aware of. https://en.wikipedia.org/wiki/Short-circuit_evaluation
Macros allow you to implement lazy evaluation in an otherwise strict language, but this is not a comprehensive explanation of their utility.
Second. I've watched pretty much all of Rich Hickey's presentations a half dozen times or more. Somewhere in the mix of more practice, another time through SICP, and watching Rich Hickey I started getting what homoiconicity means and the value prop of Lisp.
It's simultaneously very prosaic and very deep. Macros are pretty simple:
1. Access to the AST of a chunk of code for arbitrary processing.
2. At compile-time.
This is not exclusive to Lisps. I think pretty much any mature language offers a library or a built-in mechanism for getting at the AST of arbitrary code in that language. The "at compile-time" part is not necessarily built in to other languages. You absolutely can have a build system that has two (or more) passes, the first building a macro expansion framework and library of macros and the second processing the rest of your source code and rewriting the AST of the code in those files to do macro expansion.
Where Lisps are different:
1. The AST of the language is represented directly in the core data structures of the language. An operation in Lisp[0] is simply a list whose first element is an operator (one of a special operator, a macro, or a function in most Lisps) and whose remaining elements are operands to that operator. This makes AST manipulation easier, because you use the same functions/abstractions/interfaces as in normal programming, because you're directly using familiar data structures.
2. Separation of reader and evaluator (interpreter or compiler or either/both). The reader transforms textual representations into in-memory representations of values and data structures. The evaluator only ever receives these in-memory representations. Thus, it's easier to do programmatic manipulation of an AST. You never have to do code-gen of emitting source code glyphs to feed into a compiler that expects text input. Once past the reader, everything is an in-memory representation, so we never have to get back to text. That said, it's trivial to get back to text if you want, because Lisps also include a printer which takes an in-memory representation and prints the glyphs that represent their readable (i.e., can be read by the reader) textual representation.
3. The evaluation model explicitly supports tagging some code as a macro to go through macro expansion before being evaluated "normally".
4. It is idiomatic to use macros to solve code re-use problems that cannot be readily handled by standard function calls.[1]
These four Lisp traits are not exclusive to Lisp, necessarily. As I mentioned above, you could build a macro expansion and code generation library that is part of a build process for any compiled language. Eval, in interpreted languages, is as expressive as Lisp macros, but is incredibly poorly supported and unergonomic in comparison to Lisp macros.
It's an ergonomic and feasibility thing more than a capability thing. That said, POSIX shell is turing complete, so you don't need anything like C# or Lisp or Haskell or anything else we've discussed to get your computation done. The reaction you might have to comparing shell scripting to Haskell is not dissimilar to the reaction an experienced Lisp programmer might have to comparing AST manipulation and code gen through a library to Lisp's macro facilities.
[0]: Clojure, specifically abstracts the idea of callable, and there are more callable things than special operators, macros, and functions.
[1]: Languages such as Haskell and OCaml continue to push the boundaries of what can be handled by "standard function calls" (which term I hesitate to use, given its imprecision). That said, given that macros allow the use of a turing-complete language to generate inputs to the evaluator, they allow you to implement any language construct that is computable without modifying the evaluation infrastructure.
That's what drives me on. That and Rich Hickey's enthusiasm and the sense that he knows something really valuable that I don't. I watch a lot of videos as well, so I can't remember who said that programming languages were user interfaces between human and machine and are an attempt to talk to machines to get them to do things. In that sense, programming languages are a lot deeper than syntax convenience or efficiency. A good language allows us to talk to machine at a much higher level, and I get from functional programmers that this is what they find so attractive.
I will contact you if I get really stuck and when I finally get it just to share it with someone. Thanks!
edit: I've heard a lot about SICP but never looked into it. I think it's time, now that I have time
Videos: https://ocw.mit.edu/courses/electrical-engineering-and-compu...
Text: https://github.com/sarabander/sicp (links to epub and html versions at the top of the README)
I don't have a computer science degree and I think that has held me back some, but I try to overcome it by always trying to learn and never thinking I should already know something or feeling like I'm an expert. I don't have the patience at my age to go back to school and get a degree. That you attempted SICP a few times is a good sign that you don't give up and speaks to your success in finally understanding.
Compared to Elixir (another language I like) - it is homoiconic but it doesn't have this feature. I can't, for example, take an arbitrary function definition and update or define it in the REPL - because all functions must be defined in a module. However, I like these modules because I think it helps enable discovery of functions - it's much easier to find all operations you can do on lists or maps in Elixir than Clojure.
Also, I wouldn't necessarily call macros more powerful than functions. They're just different, and they enable different things. You cannot, for example, pass a macro around like you can a function. Personally I've never written them, but I use them every day because the core library defines macros. I view them as ways to let people smarter than me to give me tools I can use to write my programs.
As far as driving you on, I don't think Clojure is the end-all be-all language. I happen to like it. I also like Elixir. I know people who really love other, more standard languages though. I don't think you need to force yourself to love or "get" Clojure, but playing around with it can expand your mind a bit, especially if you've never used an FP before.
But if you want an FP that (IMHO) would be easier to transition to, Erlang was the first FP that I "got", afterwhich I tried Clojure again. The syntax is a lot more like traditional imperative programs which can ease the transition. Nowadays I would recommend Elixir over Erlang for several reasons: more modern with modern tooling, less syntax quirks, more active online community, lots of active development.
For me, the main benefits of Clojure are (other FP languages have some of these too):
1. Sharable, immutable state as the default.
2. A set of higher order functions that can do all sorts of data transformation.
3. Being able to use the same language on the web in client + server (without having to use Javascript).
4. Clojure gives me the ability to, more than other languages I've used, target the level of abstraction that I need for the problem at hand, which removes a lot of cruft.
5. Lack of special syntax: helps enable REPL driven development, makes it easier to seamlessly adopt new features from other languages.
6. It's hosted on two ecosystems I'm very familiar with - Java and Javascript.
A Macro just lets you write a function that will be executed by compiler to manipulate such tree, meaning that you can have equivalent of adding syntax like "with/using" from some languages by writing a function that takes as arguments the object and a block of code, then re-emits the same block of code but adding object.Dispose() call at the end. Or whatever else you need.
But they are still normal functions - it's the time of execution that changes.
I think I'll try to get a new understanding of Lisp macros by trying to come up with a language feature in C# or Dart that I really want but can't implement without a lot of non-intuitive difficulty, but could easily in a Lisp style macro that the compiler could use to build in a new feature at compile time.
At the same time, when you really need a macro, you really, really need a macro because it can do things a function cannot like controlling when or if its arguments are evaluated.
There are multiple reasons to use a macro rather than a function. I won't describe them here* but suffice it to say that the statement "macros are no longer useful in modern programming" is completely false.
*See pg's _On Lisp_ for an in-depth discussion of macros.
What you describe is actually used in a lot of modern software, usually self-modifying code refers to code that modifies itself while running, typically in a loop, not just generating fresh code :)
That's why I stopped doing it, but I totally get that in low level programming that it comes in very handy in some circumstances.
As someone who has given similar advice when teaching metaprogramming in the past, I'll add some context to that: most people new to metaprogramming seem to reach for it all the time for some reason, in situations where the usual programming constructs are perfectly adequate. Imagine if any time you introduce people to multithreading, they start writing all their code as parallel threaded code (which happens to some degree, but is limited by factors include what a PITA it is). That's the context in which such warnings are deemed necessary.
Just like multithreading, manual memory management, and other great-power-great-responsibility programming concepts, macros shouldn't be the first tool you reach for, but when you need them you really need them, and they can make your life so much easier.
(first-working
(get-it-from-the-cache)
(get-it-from-the-database)
(get-it-from-an-external-api)
(compute-it-the-slow-way)
default-value)
for f in GetFromCache, GetFromDatabase, GetFromAPI, Compute:
try:
return f()
except Exception as ex:
log(ex)
return default_value
def atdt_exception_suppression(*list_of_fns):
...
But macros are a powerful option because it doesn't matter if functions are first class or not. If you are using a language where functions aren't first class objects, a macro system would let Zak implement anyway.
So you are correctly identifying that Python can implement this in an alternate way as a function, but a macro system is sufficient to do this and a whole batch of other tricks even if first class functions aren't available as a feature.
def get_from_cache
get_from_cache!
rescue NoRecord
nil
end
def record
@record ||= get_from_cache || get_from_database || get_from_api || compute || default_value
end
Assume, for purposes of the exercise that someone else wrote some of the methods you're calling and they do not behave this way. You could write your own wrappers, but that's a lot more code to write and maintain than a simple control structure that doesn't care what's inside it.
With macros, I can write a short macro regardless of how the upstream code behaves. Without macros, I can request a change to the upstream code and hope the maintainer will agree to make it behave how I want.
It is also arguable that overuse of macros leads to less readable code since you wind up inventing your own personal language the more you do it, which requires more of a learning curve to work on it.
There's a large advantage to sticking with verbose idioms that are common and built out of a relatively few building blocks rather than building a meta-language which is terse.
At the same time I'm not saying don't have macros, but they should really be reserved for more extensive heavy lifting, not a pile of tiny ones used all over the codebase to make it terse.
A well written codebase should read more like Hemmingway at an 8th grade reading level using simple constructs from the base language which are put together, generally using programming design idioms which are common and terminology which everyone shares. There shouldn't be a lot of "what does this macro do and where it is it defined?" questions on every other line.
In Common Lisp:
(defmacro return-first-working (&body body)
"Expects a sequence of expressions as body.
Returns the value of the first expression of body
which returns without error."
(when body
`(handler-case ,(first body)
(error ()
(return-first-working ,@(rest body))))))
(return-first-working
(error "hi")
(error "there")
(error "!")
"hello-world"
(error "wtf?"))
Above is a recursive macro, which takes its subexpressions and returns a simplified expression, reusing itself. That way code transformation itself is described as a recursive process. One can learn how to apply such transformation patterns to the code itself and then it gets relatively easy to write.
(first-working
(get-it-from-the-cache (or local-cache default-cache))
(get-it-from-the-database current-database-connection)
(get-it-from-an-external-api (ask-user-for-api-credentials))
(compute-it-the-slow-way foo bar baz)
default-value)
There's also an advantage in clarity because an abstraction can be defined. Once you know what first-working is for, the purpose of this block of code is clear as soon as you read the first token. Using a pattern instead of an abstraction, it's necessary to read the whole loop to make sure it's really the pattern you think it is.
firstWorking() ?? getItFromTheCache() ?? getItFromTheDataBase() ?? getItFromAnExternalApi() ?? computeItTheSlowWay() ?? defaultValue;
Perl OR short-circuit:
firstWorking() or getItFromTheCache() or getItFromTheDataBase() or getItFromAnExternalApi() or computeItTheSlowWay() or defaultValue;
cache.get
.orElse(db.get)
.orElse(api.get)
... const firstSuccessfulOptionValue = fnReturnsOption
.orElse(fn2ReturnsOption())
.orElse(fn3ReturnsOption())
.orElse(fn4ReturnsOption())
.getOrElse("val");
(or a b)
Macros can analyse/manipulate the abstract syntax tree of their inputs, so I think of them as extending the compiler.
Now you only need to define the function OR...
1. Clojure does not have all of its various conditionals as language built-ins. There is a single conditional construct built in to the language, if. The rest are all defined in terms of if. Function arguments are eagerly evaluated in Clojure. How do you write a short-circuiting function in Clojure? (when predicate consequent) as a function must evaluate both the predicate and the consequent. If consequent is slow to calculate then your whole when expression is as slow as consequent, even if predicate is false and we do not need consequent. when is a Clojure macro based on the if special form.
2. core.async is a mere library.[0] This implements CSP (same idea as Go's channels) and async programming (the sort of stuff that needs compiler support in other languages such as C# or Go) as a library. If Rich Hickey didn't write core.async, you could build this control flow yourself for async programming. In fact, you can create arbitrary control flow mechanisms using macros. Common Lisp's entire object system can be implemented in macros.
3. The only special forms in Clojure are def, if, fn, let, loop, recur, do, new, ., throw, try, set!, quote, and var. The rest of what you would consider the core of Clojure consists of functions and macros. Much of the control flow functionality is implemented in macros.[1]
The core of macros is that they allow you to evaluate arbitrary user code at compile time. Clojure code is represented as native Clojure data structures (the same is true for all Lisps). The entire standard library which you are accustomed to using to manipulate standard data structures is available at compile time to transform the data structures that represent your code. I can't explain this better than Rich Hickey does in [1], so I encourage you to watch the linked section of that talk.
[0]: https://youtu.be/yJxFPoxqzWE?t=560 See this video for an overview of core.async. Later on he talks about the stuff that is implemented as macros in the library. This link starts up when he's talking about C# style async.
[1]: https://youtu.be/P76Vbsk_3J0?t=1333 See this video for an overview of the core of Clojure. The link starts with an overview of Clojure's data types and data structures (skipping the intro for the reasons behind Clojure). This leads directly into the evaluation model, which is based on the data structures of Clojure. Watch through ~1h:30m to get an overview of what macros are used for in Clojure's core.