Hard to believe this does not mention Haskell or functional programming ... but then again that was a Pycon talk from 2003.
By giving you more control over evaluation order, Haskell goes a long way toward giving you functions powerful enough to replace the macros you would've written in Lisp. That seems like the right approach to me.
However, I am now truly convinced DSLs are the future of programming.
Alan Perlis said "Beware of the Turing tar pit in which everything is possible but nothing of interest is easy". With Turing completeness, proving things (formal methods) is really hard.
If you have custom DSLs with restricted and well-understood semantics, formal methods become tractable and practical. That's the opposite of the Turing tar pit Perlis warned us about.
I imagine in the future we will have languages similar to Racket, which allow creating DSLs, and lots of mature tooling to prove things on code written using said DSLs.
Macros should not be abused, but having macros is a huge leap forward in terms of expressiveness.
Macros don't exist at runtime, functions do. You can't pass a macro to a function nor return one. You can't store a list of macros to apply to a value. Macros are nothing more than user-programmable syntactic sugar.
In a language like Haskell, you can do with functions most things that would only exist as macros in Lisp.
But by moving that transformation to compile time, you get not only the power to create syntactic sugar for the programmer but also to better control what the compiler will generate (and cache it, so you'll be free from the asymptotic complexity of the runtime control flow). That's great for both AoT compilation, for which you can effectively remove parts of the algorithm that can be pre-computed from the output program, and JIT for which runtime and compile time can interact to more intelligently distribute the workload.
You get that for free with a lazy language. You can write any sort of control flow mechanism to your heart's content and it's just a function in Haskell. People have written all manner of fancy coroutine systems, continuations, etc, without the need for macros.
moving that transformation to compile time
Haskell, gives you a great deal of control over this. Some of the best libraries make use of this functionality for things like stream fusion. Additionally, Haskell can lift constants to the top level so they're only evaluated once, on demand. This gives you free memoization without increasing code size (as you would if you precomputed a large constant).
The advantage of macros is just that they are simple to understand and use (for most you basically just have to understand tree data types, or just cons, especially for CL style macros), to implement (since every (?) language already have an AST representation, and particularly powerful in Lisp for obvious reasons) and the language doesn't even really have to support them in their normal syntax (fully separating runtime programming syntax and compile time programming syntax if they want). And for that they give a lot of power by allowing easy language extensions, syntactic sugar and powerful optimizations.
But yeah functions can pretty much describe DSLs, most of the time DSLs are written to build a declarative representation of some concept, that can easily be achieved with factory-like functions
For example LINQ in C# lets you build an AST and evaluate it at runtime for database-like languages. I think that's a good direction to go.
The problems may not look anything like the problems you can solve with a general purpose language. If they did, you wouldn't need a DSL.
Of course you need to understand your domain really really well to do this with any success.
It is true that DSL's need to be documented and maintainable, but if you have the kind of problem when a DSL might be a solution, then not having a DSL would just express the inherent complexity in a different way.
To put it another way, a clear abstraction layer (component, API, framework, DSL etc.) needs to be well-designed, documentated and maintainable, but this need is not avoided by having a system of the same complexity without such clear abstraction layers.
Btw. I disagree about the popularity of general purpose languages. Some very popular languages (VB, JavaScript) are quite badly designed but a popular due to integration with a platform.
Also DSLs still can be written without macros, a lot of frameworks use functions and builders to define a symbolic representation to be consumed later.
Today's languages don't scale up or down very well. For example, you can't write cache-optimized algorithms in Ruby and you can't write a sleek ORM in C.
Languages like Rust and C++ try to bridge the worlds by starting out being "near the metal" and then building high-level primitives on top, and that's a good start, but even with Rust's macro system — which does allow you to write some DSL-like stuff — you're always targeting Rust syntax.
Neither is every going to let you operate in a region without type annotations, for example, even though it's more productive for the programmer when experimenting/sketching, prototyping or in some contexts like shells or game scripting.
As an example, think about how much boilerplate there is around writing tests. Even though Go has a test framework built in, and tests can be nice and small, you still end up writing a lot of this:
func TestUppercase(t testing.T) {
v := Uppercase("hello")
if v != "HELLO" {
t.Errorf("expected HELLO, got %q", v)
}
}
test "Uppercase" {
Uppercase("hello") => "HELLO"
}
vertexShader : Shader { position:Vec3, coord:Vec3 } { u | view:Mat4 } { vcoord:Vec2 }
vertexShader = [glsl|
attribute vec3 position;
attribute vec3 coord;
uniform mat4 view;
varying vec2 vcoord;
void main () {
gl_Position = view * vec4(position, 1.0);
vcoord = coord.xy;
}
|]
In fact, monads have better afordance on domain specific semantics, while macros usually afford specific syntaxes. That makes macro based DSLs shallow and hard to understand when compared to the monad ones.
https://blog.rust-lang.org/2019/01/17/Rust-1.32.0.html
Building useful utilities like this within a programming language is really only possible with macros.
Not be that great is in the very nature of anything that can be "quality measurable" by humans. If "quality measurable" was or will be ever a thing
Haskell's from 1990 - it's older than Python.
I learned it because of group projects where we could pick the implementation language and that's what our group wanted.
At that time, Simon Peyton Jones was employed by Microsoft and, at least as I perceived it, was allowed to work on ghc full time because of the goodwill it generated amongst programmers for Microsoft to be the patron of Haskell.
With the advent of Learn You a Haskell, Real World Haskell, Haskell Book...etc, I really feel it is a lot more popular now, but I don't have your experience to draw upon so maybe it is just cognitive dissonance on my part.
I doubt Paul Graham was into the theoretical programming scene in that era. He might have been aware of it, but I doubt it was as production worthy as Common Lisp and thus probably not worthy of his time.
Also, Microsoft is quite open in its support for more heterogenous programming than typical Unix environment, with significant use of "niche" programming languages in various places (including things as atypical as putting custom Prolog implementation into Windows network configuration tools, just to run a prolog program that handled said configuration)