Hacker News Reader

Unfortunately, I think perhaps might have come to a different conclusion if you'd tried Racket's so-called "hygienic" macros in anger. I've found them difficult to properly understand, not very usable in practice, and overly complex for non-trivial cases.

This is not what you'll usually read in forums, I think DEFMACRO is the better design here.

Racket's macro system is quite baroque and supports a lot of things. You don't need to use hygienic "syntax-rules" at all. Consider writing your own transformer [1], which can be an arbitrary lambda function without restriction.

Or, import defmacro [2] and use that.

As can be seen, Racket actually provides more power and choice in this regard, including support for Common Lisp's way.

In any case, I make no claim to either being better or worse. I just think Racket is more principled in its mechanisms for defining and delivering a DSL to a user.

[1] https://docs.racket-lang.org/guide/proc-macros.html

[2] https://docs.racket-lang.org/compatibility/defmacro.html

Agree with "Racket is more principled..." Where I differ with you if that I'd say that Racket theorically provides more power, but the system is not tractable in practice. I suspect all large Racket programs were written by people with PhD's from Northwestern.

BTW, Racket's defmacro is only partially compatible with Common Lisp defmacro. It doesn't really work. I even seem to remember (? from long ago) that it's impossible to implement a genuine CL defmacro in Racket.

> all large Racket programs were written by people with PhD's from Northwestern

not totally wrong but i did write this https://github.com/disconcision/fructure with ~2 years programming experience. implementation features some nice simple define-syntax-rule macros which i maybe never wouldve managed with defmacro

You are hanging out in the wrong forums :-)

The `defmacro` model is "simpler", true - but it is fragile and breaks down when you want to do complicated things.

Check this blog post written by someone, who knows hos Common Lisp.

EVAL-WHEN considered harmful to your mental health https://fare.livejournal.com/146698.html

More to the point of how simple (and easy) defmacro is compared to syntax-case and syntax-rules, I like another fare post: https://fare.livejournal.com/189741.html If you're as smart as fare, it "isn't too hard to translate it".

Though I concede this case could be analogous to trying to write a Doubly Linked List in Rust, it's not enough to say definitively yea or nay. So I'd more like to see a concrete case you have in mind where the tradeoffs are squarely against CL. Like, the loop macro is more complicated than any I have written myself, but you can break it down, and it's not that bad -- I think Norvig's version is pretty neat to study: https://norvig.com/paip/loop.lisp This project (and it's not the only one!) adding C syntax to CL https://github.com/y2q-actionman/with-c-syntax I think is more complicated than loop, and is sort of where I'd put the level of "complicated things" at that I'd like to see an example from the Scheme world that clearly shows defmacro's deficiencies on some metrics. (Fewer bugs? Easier to add new features to? Shorter code? Faster performance either at compile time or runtime or both? Easier to understand or faster to implement for people with similar levels of skill in the language?)

The final `syntax-case` version:

    (define-syntax (nest stx)
      (syntax-case stx ()
        ((nest outer ... inner)
         (foldr (lambda (o i)
                  (with-syntax (((outer ...) o)
                                 (inner i))
                    #'(outer ... inner)))
                #'inner (syntax->list #'(outer ...))))))

looks fine to me. Using #` and #, and defining `reduce` one could make a version that looks like the Common Lisp implementation.

Note that the `nest` form just rearranges its input - so there is no problems getting the scope of identifiers correct. That is: we are well within the comfort zone of `defmacro`.

With regards to the `syntax-rules` solution: Fare refers to "when I learned Scheme" and pre-R6RS there were no `syntax-case` in the specification. In practise all implemenations did have macro systems more expressive than `syntax-rules`, but it was fun to see whether something was possible with plain `syntax-rules`.

Right, it's fine, and is a pretty basic macro. Doubly linked lists are pretty basic data structures too, even the Rust versions once you figure it out. I like your sibling comment making it look like the CL version. I still want to know in more detail though why you think that doing things this way instead of the CL way is less likely to be "fragile and break down" for the complicated stuff, it would help to have a specific complicated example to showcase. Perhaps the linked https://github.com/disconcision/fructure in another comment would be a good study? The author there claimed they might not have been able to manage with defmacro, maybe someone familiar with both could articulate the challenges in detail. Is it just an issue of some things benefit a lot from pattern matching, and if so, does using CL's Trivia system mitigate that at all (in the same way that using gensym+packages+Lisp-2ness can mitigate hygiene issues)? Or is it more when there's a point of complexity where you need to do code walking, and this is more straightforward in something like Racket?

It's not about pattern matching: you can use pattern matching to destructure the input of a syntax transformer (macro) in Common Lisp too. Using pattern matching makes the macros easier to read though [but that is of course a matter of opinion].

The "special sauce" is being explicit about compile time vs runtime (aka using phases).

The paper "Composable and Compilable Macros: You Want it When?" by Matthew Flatt describes why conflating runtime and compile time is a problem.

http://www.cs.utah.edu/plt/publications/macromod.pdf

Here is a version that follows the Common Lisp solution. Since `reduce` is the standard library of Common Lisp, I implemented the case I needed.

    #lang racket
    
    #;(defmacro nest (&rest r)
        (reduce (lambda (o i) `(,@o ,i)) r :from-end t))
    
    (begin-for-syntax
      (require racket/match)
      (define (reduce f xs)
        (match xs
          [(list x0 x1) (f x0 x1)]
          [(cons a d)   (f a (reduce f d))])))
    
    (define-syntax (nest r)  
      (reduce (λ (o i) #`(#,@o #,i))
              (cdr (syntax->list r))))
    
    (nest
     (list 'foo)
     (list 'bar)
     (list 'baz)
     'qux)

To each his own, obviously, but I've programmed a fair amount in CL and Scheme and vastly prefer syntax-case to defmacro. There is a little more cognitive overhead, but it all makes sense. Syntax-case macros are easier to write and read, from my point of view.

And Racket has syntax-parse, which is even nicer, I seem to recall.

> I use Common Lisp and I develop relatively large DSLs in the language, and even if I want to, it's hard—if not impossible—to write large DSLs that have nice error messages, source locations, etc. without writing an entirely new "reader" (lisp's term for a code parser) myself every time.

We have to go deeper! Could you write a DSL in CL to help you write new DSLs as easily as in Racket?

I have a project to do that, https://github.com/ruricolist/vernacular, although I'm not working on it right now.

This is what Racket people have done to write Typed Racket in it. I think it is called Turnstile: https://docs.racket-lang.org/turnstile/.

I don't think typed racket uses turnstile? At least, typed racket is far older than turnstile is.

Huh, I was under the impression, that it was based on turnstile and was sure, that I saw that somewhere, but now I cannot find any indication of it being the case. So maybe I am mistaken.

Nevermind then, turnstile is still something fitting for this topic : )

You are correct; typed racket does not use turnstile.

I’ve never used Racket, but reading your description of the documentation (and Scribble) makes me want to check out that aspect of it. I wonder how it compares with rustdoc? I do mostly Rust these days and its generated documentation is usually really good.

The generated Rust docs... are not very good. And on more complex types you have a ton of traits, with methods and all their lifetimes and generic parameters which are really hard to follow if you're unfamiliar with a library or Rust in general. Also, even the biggest libraries (clap for example) reserve a couple paragraphs for the front page prose and leave the type docs to speak for themselves... which is just not good enough.

I would go as far as to say that a non-negligible part of why people are intimidated by Rust is the rustdoc format and layout. Thankfully it makes up for it with an incredible Rust Book.

To me the gold standard of docs is Elixir followed by Racket. Even Go's docs are better than Rust, but that's mostly because of a much simpler type system, so simpler autogenerated docs for structs and their methods.

I really got annoyed by the Go package docs after I programmed in Go regularly for about a year. They were just so artless and flat. Each one is just a big list of all the stuff in the package, with no indication that the code has any structure to it at all. It was a few years ago though; perhaps it has gotten better?

Scribble beats rustdoc hands–down when it comes to authoring documentation, but even with Rustdoc you can do a good job of it if you try (http://db48x.net/reposurgeon/rust-port-docs/reposurgeon/path...). I’ll have to check out Elixir next.

Go docs are pretty terse and lifeless, agreed.

Here's an example of Elixir docs: https://hexdocs.pm/ecto/Ecto.html

Note the layout, the amount of prose to introduce a module and the function documentation that is not limited to just a quick line but often has multiple examples and tries quite hard to be clear and extensive.

There is no reason Rust docs couldn't be that good, but the example has to come from the top: improve the std docs, and everything will follow suit.

ecto and phoenix are shining examples of well written docs but its hardly the standard for elixir as a whole. many of the docs for most libraries are just lists of functions. some might have a short description though you're usually relying on a dialyzer definition at best.

Fair enough, but neither Ecto nor Phoenix are part of the stdlib, but being written by the some of core language maintainers with such a focus on documentation, they provide a great example for everybody to follow. Having excellent docs on the core libraries, the main SQL datamapper and web framework is the reason why Elixir is so loved even though it's quite a niche language.

As a new Rust fan I would just love to have the std docs half as good as Elixir's. I constantly get lost in rustdoc.

> Each one is just a big list of all the stuff in the package, with no indication that the code has any structure to it at all.

I'm afraid, much Go code doesn't have that much structure. At least none that you could express in a machine readable form.

Oh, I see. Scribble looks a little like TeX, but actually it’s an alternate read syntax/dsl. The first example where they show off the real utility of it is:

    @tabular[#:sep @hspace[1]
             (list (list @bold{Animal} @bold{Food})
                   (list "mouse"       "cookie")
                   (list "moose"       "muffin"))]

which really drops it in your lap. “tabular” isn’t from some fixed list of keywords, it is an ordinary racket function that you can call, with ordinary arguments that can be strings, lists, etc. Presumably there will be an example in the next page or two with @define[…] in it to close the loop.

And the table of contents promises literate programming as well. Someone was really thinking when they put this together; I really like it.

The documentation generated for individual functions (such as <https://docs.racket-lang.org/scribble/base.html#(def._((lib....>) is fairly nice as well. Readable, linked together by hyperlinks on the type predicates as well as hyperlinks in the body of the documentation itself; it’s pretty good. Rustdoc puts links to the source code for each one though, and I would soon miss that.

See Scribe from the end 70s: https://en.wikipedia.org/wiki/Scribe_(markup_language)

Good point; it looks like Scribe even more than TeX.

The only other equal I've seen to scribble around is Python's Sphinx. I use Sphinx for all my projects no matter what language they're in because it's so good.

I've loved using Racket docs while working through HTDP. Very clear pleasant to search/look through. Great definitions both with formal specification as well as practical examples, and all the types etc are nicely hperlinked together.

As someone trying to understand what people find special about Lisp, this was helpful. I get it now about macros, but doesn't modifying the language in the compiler phase just complicate solving the actual runtime problem? I feel I would get distracted into writing my own DSL. I've never wanted to modify the language as I was writing in it. Now I'm curious.

Don't think of it as modifying the language. Think of it as adding new ways of expressing ideas in the existing language.

These sorts of languages/language extensions pop up all the time in "ordinary" programming. For example, common DSLs we see often.

- regex is a language for string patterns

- JSON is a language for simple nested data

- SQL is a language for expressing DB queries

- OpenMP contains (essentially) a language for parallelizing loops

- (this might sound crazy because it's so common and basically built in standard in almost all programming languages:) infix notation for math

The problem is that many of these are absolute kludges in existing languages. Anybody who has had the displeasure of trying to interpolate DB table names into SQL SELECT strings knows this. Most languages try to make it a bit better by adding preprocessors, using builder-patterns, abusing reflection, or otherwise. Looking under the hood of some languages' implementations of these ideas will often lead you to see the most cursed of programmer things in existence.

Each of those languages (or close syntactic cousins of them) can be written directly in Lisp, as a library, without shoving them into strings, or using external tools, or making custom file formats. They call can exist as a part of the Lisp language as a syntax extension.

Making DSLs requires exercising that muscle a bit; it's non-trivial to design a new and useful language that fits orthogonally into an existing system.

It's like being introduced to OO programming where, at least for me, it made sense but it wasn't immediately obvious what a good object model for a problem might be. It took me lots and trial and error to understand how OO works in "the real world", where it's good, and where it falls short.

Once you get they hang of making DSLs—or even just small but powerful syntax extensions—it's an amazing tool to have at your disposal.

My second thought, which I excluded for brevity, was aren't functions and libraries language extensions? I mean technically you could write C without malloc, but it wouldn't be very useful. Regex is a great example of an embedded DSL, but I've never thought "I wish I could write a macro to change it's behavior on the fly in this special case". The last time I looked at this issue I saw that sure, if you don't have generics, it would be helpful to have a language that could write typed functions for you. That makes sense. Changing the parser and compiler behavior before any runtime code executes could be very interesting. There are certainly enough people waxing poetic about the expressiveness of it to make it worth learning.

Again, you don't want to change regex, you want to create it in the first place.

Maybe a simpler example, what if I wanted this in (pseudo-)C?

    with(s = fopen("/foo/bar", ...)) {
        ...;
        ...;
        println(readln(s));
        ...;
        ...;
     }

When the "with" clause ends, the file will automatically be closed. This is more or less bog standard RAII expressed as a new language construct.

In Lisp, this is a feature you could build and provide as a library feature. In fact, we can do it, straight here in an HN comment.

    (defgeneric destroy (x)
      (:documentation "A destructor for use with the WITH feature".))

    (defmacro with ((var = val) &body body)
      "Python-style RAII macro. VAR will be
       bound to the value of VAL, and upon
       exiting scope, will be destroyed with
       DESTROY."
      (assert (eq = ':=))
      (let ((orig (gensym)))
        `(let* ((,orig ,val)
                (,var ,orig))
           (unwind-protect
               (progn ,@body)
             (destroy ,orig))))

We can register a new destructor easily for open file streams:

    (defmethod destroy ((s stream))
      (close stream))

(We could add some more error handling, etc. to make it more robust.)

Now, we effectively have RAII in Lisp:

    (with (s := (open "/foo/bar" ...))
      ...
      ...
      (write-line (read-line s))
      ...
      ...)

This is now real Lisp code you can actually run, completely integrated into the language, and nicely orthogonal to the existing gamut of features.

This is a concept we can use almost immediately after learning about it, instead of waiting for Python's PEP 6363 to pass with consensus from people who may have different programming goals than you do.

I can see a lot of uses for macros, but in this case C# has a "using" keyword and the designers have put a lot of thought into it and how it is called in a wide variety of situations. Lisp was a language developed in a world when people worked on their own cars, like me. I would have loved Lisp if I had found it earlier, like about the time I rebuilt my VW engine in the 80's.

Many of the language simplifications on more recent C# versions, code generators, Rosylin analysers, expression trees are all features that in Lisp boil down to one thing, macros.

https://github.com/norvig/paip-lisp - Peter Norvig's Paradigm's of AI Programming

https://github.com/norvig/paip-lisp/search?l=Markdown&q=defm... - all references to defmacro in the markdown files

Chapter 3 shows a simple macro, just adding a while loop to the language.

Chapter 9 shows some more complex ones, including a with- macro and a grammar compiler macro.

Chapters 11 and 12 show the development of a Prolog implementation in CL using defmacro to aid in compilation again in Chapter 11.

Chapter 12 shows adding an OO system to the language. Technically not needed with CLOS, but a good demonstration of what can be done with macros.

There are other examples (why I included that search link). Macros let you change the language in ways large and small. Many uses could probably be replaced with functions, though you'd end up having to throw a bunch of quotes about or closures in order to delay processing things. You'd also be delaying that to runtime, which incurs its own penalties compared to macro expansion and compile time (if a compiled CL).

I like this but it seems like there would be many inconsistencies to building your own language without an actual language specification

So, as a designer of miniature languages and syntactic extensions, specify it!

A well written function must also abide by some promise or specification—contracts on the inputs and invariants on the outputs. I don't see why new syntactic structures are much different from that.

I think there's undue spookiness around macros among lots of programmers, especially those who haven't sought to solve some problem with them.

- "It's too powerful for the common person."

- "It's way too likely to make code confusing."

- "It's invariably too hard to reason about."

- "It won't work on a team because everybody is going to make their own weird incompatible language and it's just not scalable."

All of these sentiments are complete bunk, and I argue are against common wisdom around other programming abstractions, like functions or classes.

Extending a language with macros isn't all that different from extending a language with your own functions. At least in principle.

And a lot of what tends to be macros in Lisp turns into functions in eg Haskell. Eg you can define your own new branching and looping constructs (like 'if' or 'while') etc as functions in Haskell, but in Lisp these need to be macros.

Unlike call-by-need evaluation, macros provide the opportunity to compute things at compile time (think lookup tables) or analyze the arguments at compile time (think DB query syntax checking).

In Lisp, you can also define branching and looping as functions, but you need to concede and have the arguments themselves be (lambda) functions.

    (defun while (c body)
      (when (funcall c)
        (funcall body)
        (while c body)))

Yes, I know that laziness doesn't take care of all macros. That's why even Haskell has macros.

Yes, if you wrap your arguments in lambdas, you can write loops like that.

Though to be as performant as the built-in constructs, you need a rather clever compiler that 'removes' the lambdas again.

I don't know how this exactly fits, but you can do crazy stuff like query a SQL db in the compiler to check your Racket program for auto generated variable names. This extends the editor tooling to highlight names which don't match to columns in the DB tables.

I wrote a post about this here[1]

[1]: http://tech.perpetua.io/2022/01/generating-sqlite-bindings-w...

What about package management? I feel Racket’s is more standardized. I get pretty confused trying to install some requested Common Lisp packages. Browsing to the package websites, there’s no commonality, and there’s even usually no information on how to install them.

https://www.quicklisp.org/beta/

Quicklisp has become the main way many (most?) people handle installing a library for Common Lisp. Under the hood, ASDF is used for defining the system and specifying how it should be compiled and such as has also been standard in the community for a very long time.

Yeah, this is correct. ASDF and Quicklisp are essentially pervasive in the Common Lisp community now.

If you see a library you want to use, like the HTTP client drakma, then you just start your Lisp and write

    (ql:quickload "drakma")

and it will download it (if needed), install it (if needed), compile it (if needed), and load it. It's really easy, and completely changed the game for Lisp about 15 years ago.

Wow, that sounds like a great feature. Wonder if any other language has similar.

https://lisp-journey.gitlab.io/blog/overview-of-documentatio...