Clojure Macros and Metaprogramming
clojure-doc.github.io
clojure-doc.github.io
I think I've written a handful of macros in that time, there is one in my current codebase in a test class, and it could probably be removed.
Macros are for extending the language. Programmers who are new to Clojure tend to pick macros up and get a bit excited and create lots of DSLs and other cute little trinkets, and then they put macros down again because they realise they don't compose.
The power of Clojure is in simple datastructures, the core functions that operate on them, composition, and host interop.
The most useful thing that I made with macros was a library for opinionated exception handling for Clojure core.async, at the end of which I realised I was using core.async a bit too much and ended up removing a bunch of it.
The product I work on today is an Enterprise toolkit for Apache Kafka (https://kpow.io) that according to Github is 97% Clojure. Data-oriented, great delivery cadence, and terrific fun to work on. Having a single language on the JVM and in the Browser is enormous leverage, certainly we feel the benefit every day.
One of my most memorable projects[1] relied heavily on macros. I knew at the time that I was using the tools I had to automate a performance optimization technique. What I didn’t know at the time was that my former (and once again future) ecosystem, JavaScript (now TypeScript), was going through a tooling revolution (of sorts, no value attached to that observation). Babel, and then countless other tools, were transforming more and more JS. Now that sort of thing is pretty much the default.
All of that is to say: macros are not a silver bullet, they’re not trivial to write/read/maintain… but goodness dealing with similar use cases in JS environments routinely makes me long for them.
The amount of esoteric knowledge required to build something similar to my 150 line library in a JS context is absolutely bonkers. Versus just using built in language features and calling runtime functionality as needed at build time.
I don’t want to overly romanticize either macros generally or Clojure specifically. But I sincerely believe a large part of the frustration with the JS ecosystem from developers’ perspective would be significantly less frustrating if manipulating the language and sharing build/runtime logic were part of the language itself.
There is one macro defined in that library. It looks like it uses Garden’s “defstyles” macro as a key component as well.
I don’t intend to contradict what you wrote by pointing this out; it was just funny to read “relied heavily on macros” only to find exactly one “defmacro” in the codebase! This further illustrates your point: macros are a great tool to have in your toolbox, and when it’s worth writing one, it can be VERY much worth writing one.
I tend to lead more on the JVM and I have talented team-mates who drive the architecture on the JS side. I love being able to contribute both front and back end feature-wise but I don't have too much depth in understanding the bigger issues and advantages/drawbacks of Clojure (and macros) in JS land.
If you have 500 developers on a java project, you tell the new person which version of java you’re using, you mention whatever libraries or frameworks you lean on heavily and you give them a walk through the subsystems, their key components and essential abstractions and then you spend time showing how the build system is setup and how to create an environment for the system. Then they start knocking out features. There’s a bit more than I’ve said but overall It’s pretty straightforward, any java-for-hire person knows the score and everyone just cracks on with it.
How do you do this when the language itself is programmable? Clojure probably avoids my concern by two facts - it embraces immutability and it only has macros, it doesnt have reader macros.
Common Lisp has neither of those - it might be common to write in an immutable style but in Common Lisp you can (setf ) til your heart’s content. You can also define reader macros - i recently followed an example to implement json parsing in the reader. I’m still equal parts impressed, amazed and horrified.
So. You can probably tell I’ve fallen for lisp in a big way, but how do you scale up to a big dev team? Much as i love lisp in my short journey so far, it seems like a super power that could only be leveraged by up to a “two pizza team”.
In general, you don't, or not at the same rate as functions.
In big orgs, macros have three worthy uses:
- cleanly separated libraries for the org, providing a common underpinning
- smaller projects with in the org
- making your language nicer
That last one shouldn't be underestimated. I can count on one hand the # of times I've written macros for work, but I use them all the time, as in calling macros others have written.
You can very much build a big company on lisp (NuBank the most available-to-mind example). But it's true that at larger team sizes there are extra coordination obstacles that must be surmounted before you get to macro heaven.
It's almost as though you're asking for 500 Michael Angelo to help paint the sistine chapel in order to do it faster.
I'm sure there is some nonsense reason why everyone understands templates just fine and the web would literally not exist without them because html is so tedious but "macros" are this mythical complicated impossible to reason about thing??!
Seriously there is a limit to how much you can trust all the secondhand garbage "knowledge" you get from reading opinions of people who claim to have tried something.
Setting the record straight:
- common lisp is fast
- common lisp is stable
- common lisp is useful! (and interactive! even impatients can get their instant gratification)
- common lisp has non-trivial libraries for various useful things
- common lisp is simple and has excellent educational resources
- common lisp is crufty but the community has such a high average education that actually the resources have aged well and hides at least as much insight and wisdom gems as the type theoretical FP world that we now put on the pedistal we used to keep lisp on.
- common lisp is huge. No need to learn it all just hack on what you need and magically you'll be able to make software that works.
If macros are such a problem then restrict their use to seniors or "tool devs" (people who go around improving dev ergonomics by extending the language is synonymous to any other DSTool)... This way you have maximum flexibility and by restricting the words you allow you can have this flexibility while you need it while also programmatically restricting your monkeys to a short enough rope that it'll be hard to hang anyone. You can stay in some Java comparible subset (Java because lispers hate on Java like vimers do emacs).
Yes I Markov chained a bunch of sentences in this comment... I am too lazy to fully engage with the "someone is wrong on the internet"-thing, anecdotally it has been my feeling that it is no longer a good way to learn something to be "wrong on the internet" nobody is cleaning up anymore... These days everyone has a job; extract from the commons and go rave while Rome burns.
What I think it comes down to though is adherence to convention. A preference hierarchy has grown together with Clojure and it goes:
1. Can it be expressed using data structures, do that. For example a map/dictionary lookup. 2. If that fails, express your more complex idea using a function. 3. If you absolutely must, write a macro.
You usually end up with very few macros except for those in the standard library, and those you end up with are more commonly of the convenience kind. One example would be the standard web routing library of the early 10's called Compojure that relied heavily on macros where my current favourite Reitit uses just data.
Re-frame has just one, and it looks like it’s used for testing. Clara-rules has eight that are used in the library, which implements a forward-chaining Rete rule engine. Core.logic has a handful, and that’s essentially “prolog inside Clojure.” So clearly those libraries get a lot of mileage out of data and functions, with just a dash of macro.
I have yet to reach for them myself, and I can’t imagine a situation where I’d need a macro instead of a function. I think they’re probably great for library developers who are smarter than me, and I’m happy to leave them to it.
That's pretty abstract but honestly its not really such an earthshattering feature. I'd argue it's pretty much necessary for any "final" programming language to be infinitely metaprogrammable (i.e. a lisp-3) but we don't have a mature language like that yet. Probably it will arrive from laskell/hisp kinda synthesis.
I saw some of the Nubank folks speak in person in NYC before the pandemic (and before the acquisition of Cognitect) and it was hands down some of the most impressive work I've encountered. Little in the way of macros, just functions and data. What they have done to scale a Datomic infrastructure is astounding.
They have quite a lot of materials online. Worth some time.
Now that the Cognitect team is essentially embedded and more resources are going into core, I imagine at scale tooling will see some enhancements in the coming years.
Otherwise, macros are actually not super hard to figure out, Lisp macros are easy to write and that means also easy to read and understand, and you've got an interactive development environment so you can quickly explore and try things out to learn how a macro behaves.
Like you said, Clojure also doesn't have reader macros, so the syntax is standard across the board, so macro usage is limited in scope to particular call sites.
Finally, it's not like you're going to have a myriad of custom macros, and generally when you have them they make things simpler and easier, not harder. So you quickly can ramp up on the common macros of a code base and become familiar with them and start to benefit from their usage as well. They'll also likely have a doc string that documents them, so it's not like you have nothing to go with to understand what they do.
just because something is easy to write doesnt mean it's easy to read and understand - case in point : look at perl.
Similar behavior implemented with text parsing, or annotation, or pre-processor, or inside the compiler, the code to implement them is a lot less tractable.
Once you know how to write macros, it becomes fairly easy to also look at the implementation of a macro and figure out what it does. Macros tend to be very self-contained and often relatively short in implementation.
May I ask what you are doing that requires 500 devs? I've never heard of a project with that many people on it succeeding.
can you explain why this is such a big concern? as far as i can tell, you can set development guidelines and employ people who will stick to the rules. it's not like you can sneak in macros and present them as functions
Clever use of Off heap datastructures aren’t going to be enough to impress. Forget thinking your custom class loader with cooperating javaAgent that manipulates the bytes of classes to dynamically add or remove instrumentation is going to impress. These things are just taken for granted.
Even a bot that identifies dead code after traversing the AST, then re-generates valid source code from the (now pruned) AST, runs the test suite then checks in the change all by itself is seen as meh after a while.
No matter how out-there an idea is, the play area is constrained by some immutable platform rules (the java memory model). So no matter how funky things get, any other java dev will still be able to reason about any code in the system.
What happens when you remove those platform constraints? What happens when you say, we started off with lisp but we’ve ended up with something quite bespoke?
i think lisp is very nicely structured so that you only allow a certain subset to be used according to seniority. if you have good development paractices at hand, then even if you have a very bespoke program it will be well documented and anything ecentric is explained
lisp is indeed very powerful, but so is c++ for example. the difference is that the syntax of lisp allows you to be neater where you want power and more conservative where you don't, far more easily
you end up with what amounts to bank python: https://calpaterson.com/bank-python.html
If someone proposed some new dogma to live by:
1. Don’t use method parameters - write some IDL instead, add some steps to your build process to auto gen some bindings from the IDL. Introducing a dependency (with its own security patch lifecycle) in the process.
2. Don’t call a method in a library - instead setup mutual authentication and transport layer security, open a port (probably add load balancing and failover routing), set a reminder to maintain ssl certs before they expire, add distributed logging collectors, add endpoint instrumentation, add service cataloguing and lookup, slow each call down by serialising then deserialising the request then repeat that serdes overhead for the response.
…
You’d rightly point out their suggestions add complexity and slow things down, what was so bad about calling a library method in the process’s address space in the first place.
A: As you can see in the linked article, they do lots of things but macros are really useful for creating new control structures. Things like threading macros (->), (and), (for), (time), etc.
It is difficult to create new control structures in languages that don't have good macros. Try implementing new flow control using functions and appreciate how hard it is.
This is why a lisp, like Clojure, can add new control structures using a library. Many languages struggle to do that and rely on some committee somewhere deciding it is a good idea.
I haven’t looked at the Clojure source but I imagine it’s similar.
I am not very familiar with other languages that implement (un)hygienic macros, is that a common problem in things like Scheme, Lisp, or even Rust, or do they all circumvent that issue with actually private/un-nameable symbols?