Yes, but with great power comes great responsibility. The problem with FORMAT and LOOP is not that they are DSL's, but rather that they are badly designed DSLs. The reasons are different in both cases.
FORMAT is badly designed because it is write-only, kind of like Perl regexps. It is very, very hard to debug a complex FORMAT string.
LOOP, by way of very stark contrast, is very readable, even more so than regular Lisp. But it is badly designed because it is chock-full of non-orthogonal constructs. For example, WITH is completely equivalent to a set of LET bindings outside the loop, so it doesn't actually add any functionality. All it does is give you a new non-lispy syntax for creating external bindings. The semantics of FOR depend on what comes after. "for x =" does something completely different from (for example) "for x in". And the list of problems with LOOP goes on and on and on.
So the problem is not that DSLs are bad, the problem is that these DSLs are bad. But since they are part of the standard, we are stuck with them.
Additionally the LOOP sees the WITH definition in standard Common Lisp. It does not see any outside LET bindings, unless the Common Lisp would provide a feature to ask the current environment. This allows the LOOP to, for example, see a type declaration and provide a default value for the common cases.
(let (a) ; a is initialized to NIL
(declare (type integer a))
....
(loop repeat 10 do (incf a))
...
a)
(loop with a fixnum ; a is initialized to 0
repeat 10 do (incf a)
finally (return a))
Similar for the LOOP name. In the same way you could argue that this is just a named BLOCK around the LOOP construct. But, again, the name inside the LOOP makes it clear that the this is the name of this LOOP construct.
Btw., the ITERATE macro works in a similar way.
Of course it would. This:
(loop with a = ...)
(let ((a ...)) (loop ...))
(funcall (loop with a = 123 return (lambda () a))) ==> 123
I think it is highly questionable whether providing implicit typed default values is actually a useful feature, but assuming for the sake of argument that it is, why not just make a general binding construct that does this? If it's useful, it should be useful (and usable) everywhere, not just inside a loop.
> you could argue that this is just a named BLOCK around the LOOP construct
Indeed you could. :-)
> the name inside the LOOP makes it clear that the this is the name of this LOOP construct.
Yes, but why is that useful? How is (loop named foo ...) any better than (block foo (loop ...)) ? And if it is better, then why not (progn named foo ...) ? (tagbody named foo ...) ?
(loop with a = ...) maybe equivalent to (let ((a ...)) (loop ...))
but
(let ((a ...)) ... (loop ...) ...) is not equivalent to (loop with a = ...)
The difference is that in the latter it is not clear to the Lisp developer that A is meant as a LOOP local variable.
That's a design choice, not bad design.
Historically LOOP was named FOR and was only a tiny part of CLISP (Conversational Lisp) in Interlisp. There CLISP had a different syntax than normal Lisp, including iteration, binding, infix operations, etc. The idea of such a LOOP macro was then brought to Maclisp, ZetaLisp and, eventually, ANSI Common Lisp.
That's because in the latter example, A is NOT a loop-local variable. If you write:
(let ((a ...)) (some-form))
then it is clear that a is local to (some-form) regardless of what (some-form) actually is. If you write:
(let ((a ...)) (some-form) (some-other-form))
then obviously it is not clear whether a is local to (some-form) or (some-other-form) or both.
But so what? If it matters that this be clear, just don't put multiple forms inside the body of your LET, and then it is clear. You don't need to design a whole new language to solve this non-existent problem.
> That's a design choice, not bad design.
It introduces a lot of additional complexity, including a syntax that is radically different from anything else in the language, and confers very little benefit. If that is not diagnostic of a bad design I can't imagine what would be.
The designers thought that a whole new language would be useful. One may disagree with it. I would have preferred something like the ITERATE macro, but it has exactly the same idea of ENCLOSING all of these LOOP clauses, but with more parentheses.
(iterate
(with a = 1)
(incf a)
(when (> a 10)
(return a)))
(loop with a = 1
do (incf a)
when (> a 10)
return a)
Yes, obviously they thought this. They were wrong.
> ITERATE
Iterate is a slight improvement over loop because at least it doesn't require a whole new editor mode to format it properly, but simply adding parens to loop doesn't really fix it. The lack of parens is the least of LOOP's problems.
I'll give you two more examples of problems with loop. This is still nowhere near an exhaustive list.
1. There is no way to iterate over a sequence. You can iterate over a list, or you can iterate over a vector, but you cannot efficiently iterate over something that is either a list or a vector. You have to write something like:
(typecase seq
(cons (loop for item in seq ...))
(vector (loop for item across seq ...)))
2. LOOP is not extensible. There are a lot of things I might want to be able to loop over, like streams, but I can't. There are a lot of control constructs I might want to embed in a loop, like with-open-file, but I can't. Instead, I have to resort to manually writing idioms like:
(loop
with stream = (open path)
with eof-marker = (gensym "EOF")
for thing = (read stream nil eof-marker)
while (not (eq thing eof-marker))
...
finally (close stream))
(And after all that it doesn't even do the right thing if the loop code signals a condition!)
> There is no way to iterate over a sequence.
With iterate, you'd do:
(iter (for item in-sequence seq) ...)
But iterate can be extended with DEFMACRO-CLAUSE.[1] I think that's why iterate added the parentheses in the first place.
[1]: https://iterate.common-lisp.dev/doc/Rolling-Your-Own.html
I'd probably have to write a whole blog post to explain why I think iterate is only a slight improvement. But the TL;DR is that IMHO if you are writing code that uses a lot of the features of iterate or loop that is an indication that you are doing something wrong.
To cite but one example: both loop and iterate include constructs for collecting values. But collecting values has nothing to do with iterating or looping. It should be a separate construct. The right way to collect values is something like:
(with-collector collect
... (collect value) ...)
(with-collector collect
(dotimes (i 100)
(if (primep i) (collect i)))
See https://github.com/rongarret/ergolib for an implementation of WITH-COLLECTOR and lots of other constructs that are IMHO the Right Way to write code.
I don't care about such rules.
Personally I have no problem using:
(loop for i below 100 when (primep i) collect i)Yes, that is clear.
> Personally I have no problem using:
Loop is fine for simple examples like this. But I am currently maintaining a code base that has LOOPs with dozens and dozens -- sometimes a few hundred -- clauses. A single LOOP can extend over multiple pages. It's a nightmare.
Note that LOOP can fail even for simple examples. Suppose I have a list of lists of numbers and I want to collect all the prime numbers. With WITH-COLLECTOR I can do this:
(with-collector collect
(dolist (l1 l)
(dolist (n l1)
(if (primep n) (collect n)))))
And if I have a tree of items which I want to walk over and collect all of the once satisfying a predicate, LOOP just doesn't handle that at all. But by separating collection from iteration it becomes trivial:
(with-collector collect
(do-tree (item tree)
(if (predicate item) (collect item))))
The standardized LOOP is not extensible. Each LOOP implementation is extensible. A common extension is to provide new loop iteration paths.
LispWorks:
CL-USER 7 > (lispworks:defloop (s-element s-elements) elt length sequence t)
(S-ELEMENT S-ELEMENTS)
CL-USER 8 > (loop for e being each s-element of "hello-world" count e)
11
CL-USER 9 > (loop for e being each s-element of '(hello world) count e)
2
https://research.gold.ac.uk/id/eprint/2344/1/sequences-20070... gives an example to extend SBCL's LOOP (which is based on the MIT LOOP implementation).
There are lots of things in the Common Lisp standard, which are not extensible, but where implementations and libraries provide extensible versions. Example: the CLOS MOP, sequences, hash tables, ...
Allegro isn't. Neither is SBCL. Nor CCL. Nor CLisp. In fact, the only implementation that offers this feature AFAICT is Lispworks.
> There are lots of things in the Common Lisp standard, which are not extensible
Sure. So? Just because loop is not the only thing in CL that sucks does not make it suck any less.
I don't see a lot of value in "X is just Y and Z, why use X?" arguments. Clearly it has enough ergonomic benefit that people like using it.
I definitely think the design of the DSL is a little chaotic, not to mention difficult to remember, and having two entirely different DSLs implemented by the same macro is downright ridiculous. But it's definitely good enough at its job and better than not having it at all. If it didn't exist, people would probably be DIYing their own (even worse) versions of it, without the assurance of it being part of the specification and ostensibly a well-tested part of any implementation.
And sharing lisp code, is that a thing people do? (this half an insult half a joke) So you can easily enforce your own ideals in your own project.
I'm not advocating getting rid of format, or even loop (which is by far the greater of the two evils). I'm pointing them out as cautionary tales for future DSL designers.
Similarly, I like LOOP more than I'm supposed to, if I go by what I read online.
An ambitious young student of Lisp came to Master Foo, seeking to deepen her knowledge and understanding.
One day, during their studies and practice, the students stopped and said, "Master, I am troubled. In Lisp we have minimal syntax, and we use S-expressions to convey structure, and this syntactic uniformity gives us great power to build our own composable syntax constructs. And yet we have LOOP in the standard, which is an ad-hoc imitation of the same arbitrary syntax of other languages that we use Lisp in order to avoid!"
The next day, the Master brought the student on a hike into the hills. As the sun was setting late in the afternoon, they reached a small cabin near the summit of a hill. A village of huts was visible in the valley below them. "Observe that this building is constructed from many small pieces of wood, each nearly uniform in size and shape. With those pieces I long ago built a hut that resembles the shoddy huts of the village, yet this one is both strong and easy to modify. You could even build it yourself from plans." The student observed and agreed, and was very impressed, but was also confused. "Master, what does this teach me about Lisp?" she asked. Master Foo continued, "You shall stay here on the mountain tonight. Return to me in the morning."
With that, the Master drew a key from his robe, firmly locked the door of the cabin, and hiked away. As he did so, a cold rain and steady wind began to blow. The student spent the night cold and damp on the ground beside the cabin, and awoke enlightened.
I prefer them to the functional versions sometimes so I don’t have to write lambda, the for loops have options for combining multiple lists, etc.
I have no doubt much of this can be done with the racket constructs. But I still find I can more easily read loop usages I've written in the past. Whereas many similarly dense for/folds are opaque within a few minutes for me. :(
I do appreciate the convenience and brevity of FORMAT as it currently exists, but I also think it would be pretty cool to have a proper S-expression macro that expands to a FORMAT control string. It would be analogous to a macro that generates SQL or HTML.
(format t (formatter "~a-~b") x y)
(format t "~a-~b" x y)
10000% agree. LOOP and FORMAT are excellent examples of using lisp to build dsls. so much so that they made it into the ANSI standard