Comment by derefr | Hacker News Reader

Comment by derefr | Hacker News Reader

Parent Full thread

Hygenic macros are still "the big problem."

There is a reason there are no thousand-developer FOSS projects written in a Lisp; and there is a reason Ruby is derided for "monkey-patching"; and there is a reason that a single parse-transform in an Erlang project is a very strong code-smell that needs intensive justification.

The reason, in all of these cases, is that a project that includes one unique macro definition, is now a project effectively written in a different language—"language Foo, plus macro Bar." HN, for example, is written in Arc, a.k.a. "Racket + several macros." Emacs is written in Emacs Lisp, a.k.a. "MacLisp + quite a few macros." In general, if a project goes on long enough without any prohibition against adding yet another macro to the codebase, eventually that project—like Arc, like Emacs—will have accidentally created an entirely-separate language.

But before then—even on the commit of the very first macro definition—the prerequisite knowledge required to contribute to the codebase has now diverged: you can't just jump in as someone who "knows Foo" and immediately understand the logical flow of 100% of the code. To contribute to such projects, people have to learn how the macros change the logical flow—i.e. learn that diverged language.

To which they very likely will say: screw that! Thus capping the number of contributors, and/or the speed of development.

> There is a reason there are no thousand-developer FOSS projects written in a Lisp

Yes; the main one being that about 996 of them wouldn't have anything to do.

> you can't just jump in as someone who "knows Foo" and immediately understand the logical flow of 100% of the code.

Can you just jump into the Linux kernel as "someone who knows C" and understand the logical flow of everything? USB, TCP/IP stack, virtual memory, file systems, block devices, scheduler, ...

Maybe I wasn't clear with my wording.

I didn't mean "logical flow" to refer to following the branches of the code at runtime by keeping a model of the code's run-time state in your head. That does, indeed, require having a pretty solid grasp of the entire codebase (and, in large projects, is just rarely done for this reason.)

Instead, I meant "logical flow" to refer to, effectively, the ability to do static analysis of the code in your head: knowing which blocks of code are live or dead; which blocks might execute zero or one times, like a branch, or zero or more times like a loop; which blocks might execute immediately and which will be closed over and then thunked later; which blocks are within synchronization barriers and which aren't, and so will be touched by multiple threads; etc.

When you know a language, and you're debugging code written in it, this is how you "skim" the code efficiently: you figure out where the Instruction Pointer might go, and then you follow it through the blocks of code it might execute along the way.

In the absence of macros, this is possible. But in the presence of macros—any of which could slice-and-dice your code, rearranging it, throwing some bits away, duplicating others, wrapping any given given expression in an if{} or a while{} or a lambda{} or a synchronize{}—you now have no idea what a given block of code that calls a macro will do. Where is the Instruction Pointer allowed to go? Who knows? It might even enter—for quite a long time—code that's only referred to in the macro body, not at the macro call-site. (But only sometimes; at other call-sites the macro-expansion might not call that function at all. It might depend on the function. It might depend on the module attributes. It might depend on the time of day!)

A block of code passed through a macro is its own world with its own rules. Those rules might be 100% similar to our own world save for one little thing... or they might be "only compile this to an expression if DEBUG has been defined at the project level" or "apply the statement body to itself as a Y-combinator" or "convert all blocking calls in your function into async calls followed by returns to the trampoline of an FSM to enable continuations, splitting your block at those points into several private functions."

Well, yes. If you don't know what defclass is, how do you know that (defclass a (b c)) isn't calling a function b with argument c?

You have to understand the macros. Either their documentation, or their code if that is lacking. If we don't know the macro at all, we can't do static or dynamic or any kind of analysis; we don't have the information.

Do you know what happens inside a function without chasing its definition, and then chasing its children and so on?

Programming is all about "controlling something with this little piece of text here, such that the real meat of it is done in multiple elsewheres". You do this with functions, or macros. If you don't do this ("everything happens right here and nowhere else") you will not get anywhere beyond a certain small complexity.

Okay, I know that a function in C is call by value. These expressions are evaluated and converted into argument values. They are not relocated into another scope, or turned into thunks or whatever. Well, so what? Once the function takes over, I have no idea, if I don't know what the function does.

> Do you know what happens inside a function without chasing its definition, and then chasing its children and so on?

Yes. That was my point: when you know a language, you can "unfocus your eyes" and read code without knowing anything about what the code does, except that A calls B calls C. You don't need to know what A/B/C are, or what they're supposed to do; if the program crashed, "errors in business logic" aren't your concern. You want to find an error in the model of the world the program is using, that lead to an "impossible" scenario. You want to uncover a series of static guarantees—or lack thereof—that lead to the possibility of a code-path [containing a throw() or an abort() or whatever] being activated that is never supposed to be activated.

If you have a crash with a backtrace of function C <- function B <- function A, then you will start with A, and look at the code around the call to B to see if any of it "is relevant"—i.e. if it might, lexically, have been the cause of the problem—and, after a second or two, you likely determine that it wasn't. So you move on to B. Do the same. Move on to C. Stare at the code. See something that smells vaguely wrong. Then start trying to understand the code, from there. Switch from skimming, to reading.

Any developer versed in a language can do this to any program written in that language, with no idea what it is they're reading. This is why "over-the-shoulder" debugging can magically work—if you know a language, you can "know broken code when you see it" without knowing a thing about the code itself.

But if, in the above, function A's call to function B, or B's call to function C, happened within a block wrapped in a macro call—now you can't skim. Now Mr. Joe "over-the-shoulder" Schmoe has no idea whether A or B are relevant. Now you have to go and find the docs/definitions of those macros, and familiarize yourself with them, before you can do anything.

If there's one macro, this is easy enough. FOSS projects of this type still have many contributors, although there might be sections of the codebase considered "harder to contribute to" than others. (See: the Linux kernel's syscall mapper macros.)

But if the whole language is effectively half-DSL—and not a business domain kind of DSL, but just the developers' idea of a bunch of awesome extra primitives, like Arc or Elisp—then random drive-by "oh, hey, I noticed that was broken and fixed it" FOSS contributions will go to zero, because nobody will spend enough time getting versed in the new one-off language to be able to "notice something was broken."

No, those would require domain specific knowledge.

But at least inside the Linux kernel everyone is talking the equivalent of English.

Understanding English does not imply you will be able to jump into any random scientific paper with 45 years of background theory and immediately understand it either.

Being able to understand the grammar and sentence structure really helps, even if you don't understand the vocabulary or technical principles of the domain you are reading.

Domain specific knowledge, domain specific language, ...

I smell fear of macros.

A macro is just another function, but it does some code transformation. The code base will not explode, the world will not stop revolving, just because a few macros.

I use LispWorks, which adds a few hundred macros (amongst many other things) to Common Lisp. It's still Common Lisp, but extended with macros for defining new data types, macros for defining UI features, even a few new control structures have been added. For most of the stuff I don't even have the source code, since it is a closed commercial product - but the macros are documented.

Lisp is still used and one of the reasons is its seamless integration of various meta-programming facilities. This requires developers to learn a slightly more complicated language. But in larger projects this is not more difficult than learning the architecture tools of large OO frameworks - frameworks which are common in Java, Javascript and many other languages.

Not fear, not at all. Resentment. I work on a codebase full of macros. I wrote a lot of them, when I was more naive. And now I'm the one who can't get any help with my code, because nobody wants to spend the time to familiarize themselves. :/

What have you done to get people interested in the code base?

Have you conducted a survey which confirms that lots of people want to help, but for the macros? Or are you guessing at the reasons?

First of all, do lots of people use it; is there a large user to contributor ratio?

It's hard to get help with a FOSS project no matter what; the ones that get help are vastly outnumbered by those which don't get help. According to openhub.net, "over half of all active projects [listed] on Open Hub are solo efforts". (This is boiler plate text which appears in the summary of projects which are that way.)

Reply on news.ycombinator.com