What are some genuinely useful things macros can do?
Without macros, regular expressions are strings at compile time and then at runtime objects are created based on the strings to evaluate them.
With macros, regular expressions are expanded at compile time into Common Lisp code that implements the evaluation.
There's a similar idea with an ORM library.
Without macros, the object-relational mapping is defined at compile time and any dynamic SQL is generated at run time.
With macros, the object-relational mapping is used at compile time to expand the query macros into Common Lisp code that includes the actual SQL.
Then also remember that the compiler is available at runtime in a lisp, so the distinction only matters when comparing it to non-lisps.
A small tidbit: you don't actually need full syntactic macros to get compile time computation in CL, compiler macros (https://www.lispworks.com/documentation/HyperSpec/Body/03_bb...) suffice.
https://github.com/nim-works/cps/blob/master/docs/cps.svg
[1] Yeah, I know, this word is extremely overworked in that sentence. Sorry, that's an exaggeration for effect. In reality, it's not that simple, and not always possible: it depends on the language and macro system. It's still more likely to be implementable in library code if there is a macro system, though.
If I want some new syntax in a LISP, I just write a macro.
e.g. the following threading macro in clojure
(->> x foo bar)
Takes x and passes it as the last argument to foo, then the result of foo is passed as the last argument to bar. Alternative you have -> for passing it as the first argument. These are from the standard library, but if they weren't they would be trivial to write yourself.
And no strings involved when you write macros in lisp either. It's just lists, your macro is just building a list. Because the syntax is just lists and there's no difference between lists and the syntax.
Macros let a domain programmer (as opposed to the compiler writer) in Lisp implement almost any other programming paradigm as a native language feature.
Racket's define-syntax/syntax-parse has escape hatches into procedural generation (through which you can easily break the hygiene), and the equivalent of CL's &environment contains more information and allows more operations in Racket. Racket also supports reader extensions but often doesn't need them because it has a macro-based API for module definitions (a feature completely absent from, or even alien to, Common Lisp; OTOH, Elixir borrows some of Racket's ideas in this space).
For a practical example, Coalton and Typed Racket implementations show that both systems are capable of large-scale language extensions. I never studied their respective codebases, so I can't say anything about relative ease of introducing those extensions, but they are possible in both cases, at least.
What Let Over Lambda calls limited power, is probably in reference narrowly to `syntax-rules`. It's not just that it's hygienic, but that it purposefully has no way to opt out of it (unhygienic capabilities in a hygienic system are trivial to provide; but they still need to be provided by language implementation) but what's even more limiting is that it doesn't let you write procedural macros, it's all arcane pattern matching and templates. `syntax-case` (part of r6rs) doesn't have these issues.
IME, the primary use case for macros in Lisps is language extension, a step before a DSL. For every `loop` there are thousands of `with-x` macro implementations. See the macro-writing macros in Alexandria/Serapeum for the most common patterns.
Also, some dynamic languages (Io, Prolog, Groovy) support DSLs through non-macro mechanisms. DSLs in those languages are as easy (or easier) to implement as in Lisps; however, they tend to be less performant. So, again, compile-time macro execution does matter for the DSL use case, too.
The characterization of it not being "compile-time macro execution" is strange, what do you think these predicates are doing?
I have some reading to do, but overall it seems like macros are favored instead of fexprs. Macros completely avoid some environment handling issues.
> Messages as Code — Messages form trees that can be inspected and rewritten at runtime. Argument evaluation can be deferred, so if, while, and for are implementable in Io itself.
Sure but you've also got CTFE in languages that lack a proper macro system such as C++. The defining characteristic (IMO) is the access to and ease of manipulating the AST on the fly. And any time you're outputting an AST you're going to need access to the compiler at which point the line between run time and compile time becomes blurred and arbitrarily nested.
You can create DSLs without macros, for example by parsing and interpreting at runtime, but macros let you move much of that logic to compile time.
For me, that's the killer feature: writing code that looks dynamic or interpreted, but ends up as compiled code.
> Clojure is actually an interesting case here, because it is hosted. So, you can write a Clojurescript macro that runs at compile time on the JVM, but emits code that runs at runtime in Javascript, e.g. instead of blindly loading and executing some JS lib code, your macro can first parse it, analyze it, and conditionally emit Cljs that changes the runtime behavior. Use-cases for code that writes code across a host boundary are rare, yet enormously useful and really difficult to achieve without homoiconic nature of the language.
> Hyperfiddle/Electric is a nice project that effectively utilizes the idea.
... to do something absolutely mind-blowing.