Hacker News Reader

>How do you use lisp style macros when you have 500 developers working on a single product?

In general, you don't, or not at the same rate as functions.

In big orgs, macros have three worthy uses:

- cleanly separated libraries for the org, providing a common underpinning

- smaller projects with in the org

- making your language nicer

That last one shouldn't be underestimated. I can count on one hand the # of times I've written macros for work, but I use them all the time, as in calling macros others have written.

You can very much build a big company on lisp (NuBank the most available-to-mind example). But it's true that at larger team sizes there are extra coordination obstacles that must be surmounted before you get to macro heaven.

NuBank promoting the fact their average employees age is 26 just makes me feel old as I approach 28 lol...

i think the crux is that it's hard to find 500 people who are all at such high level of capability that could make use of macros in lisp - esp. if they all need to follow a similar style of programming (iirc, lisp macros tend to follow idiosyncrasies of the person writing it).

It's almost as though you're asking for 500 Michael Angelo to help paint the sistine chapel in order to do it faster.

Bah. This meme vastly overstate the difficulty...

I'm sure there is some nonsense reason why everyone understands templates just fine and the web would literally not exist without them because html is so tedious but "macros" are this mythical complicated impossible to reason about thing??!

Seriously there is a limit to how much you can trust all the secondhand garbage "knowledge" you get from reading opinions of people who claim to have tried something.

Setting the record straight:

- common lisp is fast

- common lisp is stable

- common lisp is useful! (and interactive! even impatients can get their instant gratification)

- common lisp has non-trivial libraries for various useful things

- common lisp is simple and has excellent educational resources

- common lisp is crufty but the community has such a high average education that actually the resources have aged well and hides at least as much insight and wisdom gems as the type theoretical FP world that we now put on the pedistal we used to keep lisp on.

- common lisp is huge. No need to learn it all just hack on what you need and magically you'll be able to make software that works.

If macros are such a problem then restrict their use to seniors or "tool devs" (people who go around improving dev ergonomics by extending the language is synonymous to any other DSTool)... This way you have maximum flexibility and by restricting the words you allow you can have this flexibility while you need it while also programmatically restricting your monkeys to a short enough rope that it'll be hard to hang anyone. You can stay in some Java comparible subset (Java because lispers hate on Java like vimers do emacs).

Yes I Markov chained a bunch of sentences in this comment... I am too lazy to fully engage with the "someone is wrong on the internet"-thing, anecdotally it has been my feeling that it is no longer a good way to learn something to be "wrong on the internet" nobody is cleaning up anymore... These days everyone has a job; extract from the commons and go rave while Rome burns.

I think the lack of reader macros really help here. Macros in Clojure are much more "functions that run at compile time" rather than "look at this fun DSL I threw together".

What I think it comes down to though is adherence to convention. A preference hierarchy has grown together with Clojure and it goes:

1. Can it be expressed using data structures, do that. For example a map/dictionary lookup. 2. If that fails, express your more complex idea using a function. 3. If you absolutely must, write a macro.

You usually end up with very few macros except for those in the standard library, and those you end up with are more commonly of the convenience kind. One example would be the standard web routing library of the early 10's called Compojure that relied heavily on macros where my current favourite Reitit uses just data.

The last time “macros in Clojure” came up, I searched through some of the libraries I’ve run across to see how often they’re used and, as you suggest, it ain’t much.

Re-frame has just one, and it looks like it’s used for testing. Clara-rules has eight that are used in the library, which implements a forward-chaining Rete rule engine. Core.logic has a handful, and that’s essentially “prolog inside Clojure.” So clearly those libraries get a lot of mileage out of data and functions, with just a dash of macro.

I have yet to reach for them myself, and I can’t imagine a situation where I’d need a macro instead of a function. I think they’re probably great for library developers who are smarter than me, and I’m happy to leave them to it.

I think they come in handy when you've got some functionality that has to happen in a context that you can't easily move between function calls... Then you benefit from the ability to template into the description of that context some actions to perform _before_ you enter into the "black box".

That's pretty abstract but honestly its not really such an earthshattering feature. I'd argue it's pretty much necessary for any "final" programming language to be infinitely metaprogrammable (i.e. a lisp-3) but we don't have a mature language like that yet. Probably it will arrive from laskell/hisp kinda synthesis.

Highly recommend checking out materials from Nubank, the Brazilian banking startup that now owns Cognitect (and therefore Clojure). They have several hundred engineers on the back end working almost entirely in Clojure. Believed to be the largest Clojure plant, though many other large orgs now have sizable Clojure staffing.

I saw some of the Nubank folks speak in person in NYC before the pandemic (and before the acquisition of Cognitect) and it was hands down some of the most impressive work I've encountered. Little in the way of macros, just functions and data. What they have done to scale a Datomic infrastructure is astounding.

They have quite a lot of materials online. Worth some time.

Now that the Cognitect team is essentially embedded and more resources are going into core, I imagine at scale tooling will see some enhancements in the coming years.

Most enterprise software nowadays is broken up in services or micro-services anyways, and monoliths spanning more than 6 to 15 developers are becoming a lot more rare. If you've got a monolith with 500 developers touching the code base you're in trouble even if you used Haskell.

Otherwise, macros are actually not super hard to figure out, Lisp macros are easy to write and that means also easy to read and understand, and you've got an interactive development environment so you can quickly explore and try things out to learn how a macro behaves.

Like you said, Clojure also doesn't have reader macros, so the syntax is standard across the board, so macro usage is limited in scope to particular call sites.

Finally, it's not like you're going to have a myriad of custom macros, and generally when you have them they make things simpler and easier, not harder. So you quickly can ramp up on the common macros of a code base and become familiar with them and start to benefit from their usage as well. They'll also likely have a doc string that documents them, so it's not like you have nothing to go with to understand what they do.

> Lisp macros are easy to write and that means also easy to read and understand

just because something is easy to write doesnt mean it's easy to read and understand - case in point : look at perl.

Fair, in this case my feelings are they are also easy to read and understand, compared to alternatives.

Similar behavior implemented with text parsing, or annotation, or pre-processor, or inside the compiler, the code to implement them is a lot less tractable.

Once you know how to write macros, it becomes fairly easy to also look at the implementation of a macro and figure out what it does. Macros tend to be very self-contained and often relatively short in implementation.

> How do you use lisp style macros when you have 500 developers working on a single product?

May I ask what you are doing that requires 500 devs? I've never heard of a project with that many people on it succeeding.

> how do you scale up to a big dev team?

can you explain why this is such a big concern? as far as i can tell, you can set development guidelines and employ people who will stick to the rules. it's not like you can sneak in macros and present them as functions

What happens when you scale java development to a team of that size? Short version: every optimisation you can dream of gets implemented.

Clever use of Off heap datastructures aren’t going to be enough to impress. Forget thinking your custom class loader with cooperating javaAgent that manipulates the bytes of classes to dynamically add or remove instrumentation is going to impress. These things are just taken for granted.

Even a bot that identifies dead code after traversing the AST, then re-generates valid source code from the (now pruned) AST, runs the test suite then checks in the change all by itself is seen as meh after a while.

No matter how out-there an idea is, the play area is constrained by some immutable platform rules (the java memory model). So no matter how funky things get, any other java dev will still be able to reason about any code in the system.

What happens when you remove those platform constraints? What happens when you say, we started off with lisp but we’ve ended up with something quite bespoke?

> What happens when you remove those platform constraints? What happens when you say, we started off with lisp but we’ve ended up with something quite bespoke?

i think lisp is very nicely structured so that you only allow a certain subset to be used according to seniority. if you have good development paractices at hand, then even if you have a very bespoke program it will be well documented and anything ecentric is explained

lisp is indeed very powerful, but so is c++ for example. the difference is that the syntax of lisp allows you to be neater where you want power and more conservative where you don't, far more easily

> What happens when you say, we started off with lisp but we’ve ended up with something quite bespoke?

you end up with what amounts to bank python: https://calpaterson.com/bank-python.html

This is the best answer so far. Bank Python ain’t so bad. I’m going to stop worrying so much about lisp macro abuse. And I’m going to finally start reading let over lambda.

the difference being of course that python the language is not designed for this, whereas lisp is

I work in an organization with thousands of engineers; the largest projects have perhaps 10 contributors. Coordination with huge numbers of developers is done through gRPC; doesn’t require compromising on service internals.

That’s a way to reduce the output of your developers - keep them feeling busy but don’t get features to customers any faster.

If someone proposed some new dogma to live by:

1. Don’t use method parameters - write some IDL instead, add some steps to your build process to auto gen some bindings from the IDL. Introducing a dependency (with its own security patch lifecycle) in the process.

2. Don’t call a method in a library - instead setup mutual authentication and transport layer security, open a port (probably add load balancing and failover routing), set a reminder to maintain ssl certs before they expire, add distributed logging collectors, add endpoint instrumentation, add service cataloguing and lookup, slow each call down by serialising then deserialising the request then repeat that serdes overhead for the response.

…

You’d rightly point out their suggestions add complexity and slow things down, what was so bad about calling a library method in the process’s address space in the first place.

With thousands of developers, we can afford platform teams that make all of those things painlessly on-by-deafault on our standard service frameworks.

Macros are linguistic abstractions; you describe them as part of your briefing on "key components and essential abstractions". In practice, most macros tend to look the same, e.g. with-<foo> macros only differ in how to acquire and release some <foo>.

Rails is basically a DSL for web applications and it scales to big teams. It might work this way for Lisp as well.