Comment by lisper | Hacker News Reader

Comment by lisper | Hacker News Reader

Parent Full thread

> isn't making a DSL just about the most lispy thing you could do?

Yes, but with great power comes great responsibility. The problem with FORMAT and LOOP is not that they are DSL's, but rather that they are badly designed DSLs. The reasons are different in both cases.

FORMAT is badly designed because it is write-only, kind of like Perl regexps. It is very, very hard to debug a complex FORMAT string.

LOOP, by way of very stark contrast, is very readable, even more so than regular Lisp. But it is badly designed because it is chock-full of non-orthogonal constructs. For example, WITH is completely equivalent to a set of LET bindings outside the loop, so it doesn't actually add any functionality. All it does is give you a new non-lispy syntax for creating external bindings. The semantics of FOR depend on what comes after. "for x =" does something completely different from (for example) "for x in". And the list of problems with LOOP goes on and on and on.

So the problem is not that DSLs are bad, the problem is that these DSLs are bad. But since they are part of the standard, we are stuck with them.

The idea is that the WITH variables are local to the LOOP construct. If one leaves the LOOP, the variables are gone. With an outside LET this would not be the case. The LOOP thus gives scope for all lexical variables inside of it.

Additionally the LOOP sees the WITH definition in standard Common Lisp. It does not see any outside LET bindings, unless the Common Lisp would provide a feature to ask the current environment. This allows the LOOP to, for example, see a type declaration and provide a default value for the common cases.

  (let (a)   ; a is initialized to NIL
    (declare (type integer a))

    ....

    (loop repeat 10 do (incf a))

    ...

    a)

vs.

  (loop with a fixnum   ; a is initialized to 0
        repeat 10 do (incf a)
        finally (return a))

It provides a compact notation with ONE scoping construct, the LOOP macro.

Similar for the LOOP name. In the same way you could argue that this is just a named BLOCK around the LOOP construct. But, again, the name inside the LOOP makes it clear that the this is the name of this LOOP construct.

Btw., the ITERATE macro works in a similar way.

> The idea is that the WITH variables are local to the LOOP construct. If one leaves the LOOP, the variables are gone. With an outside LET this would not be the case.

Of course it would. This:

    (loop with a = ...)

is exactly equivalent to:

    (let ((a ...)) (loop ...))

and so you can e.g. capture with-bindings in closures:

    (funcall (loop with a = 123 return (lambda () a))) ==> 123

> Additionally the LOOP sees the WITH definition in standard Common Lisp. It does not see any outside LET bindings, unless the Common Lisp would provide a feature to ask the current environment. This allows the LOOP to for example see a type declaration and provide a default value for the common cases.

I think it is highly questionable whether providing implicit typed default values is actually a useful feature, but assuming for the sake of argument that it is, why not just make a general binding construct that does this? If it's useful, it should be useful (and usable) everywhere, not just inside a loop.

> you could argue that this is just a named BLOCK around the LOOP construct

Indeed you could. :-)

> the name inside the LOOP makes it clear that the this is the name of this LOOP construct.

Yes, but why is that useful? How is (loop named foo ...) any better than (block foo (loop ...)) ? And if it is better, then why not (progn named foo ...) ? (tagbody named foo ...) ?

As I said LOOP is designed such that names, local variables, local iteration variables are all defined INSIDE the LOOP, such that LOOP can process them and that they are all using the LOOP syntax.

(loop with a = ...) maybe equivalent to (let ((a ...)) (loop ...))

but

(let ((a ...)) ... (loop ...) ...) is not equivalent to (loop with a = ...)

The difference is that in the latter it is not clear to the Lisp developer that A is meant as a LOOP local variable.

That's a design choice, not bad design.

Historically LOOP was named FOR and was only a tiny part of CLISP (Conversational Lisp) in Interlisp. There CLISP had a different syntax than normal Lisp, including iteration, binding, infix operations, etc. The idea of such a LOOP macro was then brought to Maclisp, ZetaLisp and, eventually, ANSI Common Lisp.

> The difference is that in the latter it is not clear to the Lisp developer that A is meant as a LOOP local variable.

That's because in the latter example, A is NOT a loop-local variable. If you write:

(let ((a ...)) (some-form))

then it is clear that a is local to (some-form) regardless of what (some-form) actually is. If you write:

(let ((a ...)) (some-form) (some-other-form))

then obviously it is not clear whether a is local to (some-form) or (some-other-form) or both.

But so what? If it matters that this be clear, just don't put multiple forms inside the body of your LET, and then it is clear. You don't need to design a whole new language to solve this non-existent problem.

> That's a design choice, not bad design.

It introduces a lot of additional complexity, including a syntax that is radically different from anything else in the language, and confers very little benefit. If that is not diagnostic of a bad design I can't imagine what would be.

> You don't need to design a whole new language to solve this non-existent problem.

The designers thought that a whole new language would be useful. One may disagree with it. I would have preferred something like the ITERATE macro, but it has exactly the same idea of ENCLOSING all of these LOOP clauses, but with more parentheses.

  (iterate
   (with a = 1)
   (incf a)
   (when (> a 10)
     (return a)))

is

  (loop with a = 1
        do (incf a)
        when (> a 10)
          return a)

If one cares about radically different syntax then you wouldn't use LOOP at all. (loop for i in l do (print i)) is equivalent to (dolist (i l) (print i)) which is equivalent to something with a LET/TAGBODY... construct. Why not just use the latter? Maybe its design choices made it practical enough?

> The designers thought that a whole new language would be useful.

Yes, obviously they thought this. They were wrong.

> ITERATE

Iterate is a slight improvement over loop because at least it doesn't require a whole new editor mode to format it properly, but simply adding parens to loop doesn't really fix it. The lack of parens is the least of LOOP's problems.

I'll give you two more examples of problems with loop. This is still nowhere near an exhaustive list.

1. There is no way to iterate over a sequence. You can iterate over a list, or you can iterate over a vector, but you cannot efficiently iterate over something that is either a list or a vector. You have to write something like:

    (typecase seq
      (cons (loop for item in seq ...))
      (vector (loop for item across seq ...)))

which forces you to duplicate all of the LOOP code twice. Worse, you have to actually duplicate all the code represented by the elipses. You can't abstract it away in a function or a macro.

2. LOOP is not extensible. There are a lot of things I might want to be able to loop over, like streams, but I can't. There are a lot of control constructs I might want to embed in a loop, like with-open-file, but I can't. Instead, I have to resort to manually writing idioms like:

    (loop
      with stream = (open path)
      with eof-marker = (gensym "EOF")
      for thing = (read stream nil eof-marker)
      while (not (eq thing eof-marker))
      ...
      finally (close stream))

and again, because of the non-orthogonal syntax, I can't abstract all that away in to a macro either. I have to write it all out manually every single time I want to iterate over the contents of a file.

(And after all that it doesn't even do the right thing if the loop code signals a condition!)

Iterate doesn't have either of these problems, so I don't see how it's just a "slight improvement".

> There is no way to iterate over a sequence.

With iterate, you'd do:

  (iter (for item in-sequence seq) ...)

> LOOP is not extensible.

But iterate can be extended with DEFMACRO-CLAUSE.[1] I think that's why iterate added the parentheses in the first place.

[1]: https://iterate.common-lisp.dev/doc/Rolling-Your-Own.html

> I don't see how it's just a "slight improvement"

I'd probably have to write a whole blog post to explain why I think iterate is only a slight improvement. But the TL;DR is that IMHO if you are writing code that uses a lot of the features of iterate or loop that is an indication that you are doing something wrong.

To cite but one example: both loop and iterate include constructs for collecting values. But collecting values has nothing to do with iterating or looping. It should be a separate construct. The right way to collect values is something like:

    (with-collector collect
      ... (collect value) ...)

Now you can collect values whether or not you are looping, and regardless of what iteration construct you decide to use. You don't need special constructs for conditional behavior. So, for example, you could do this:

    (with-collector collect
      (dotimes (i 100)
        (if (primep i) (collect i)))

to get a list of primes under 100.

See https://github.com/rongarret/ergolib for an implementation of WITH-COLLECTOR and lots of other constructs that are IMHO the Right Way to write code.

> But collecting values has nothing to do with iterating or looping. It should be a separate construct.

I don't care about such rules.

Personally I have no problem using:

  (loop for i below 100 when (primep i) collect i)

> I don't care about such rules.

Yes, that is clear.

> Personally I have no problem using:

Loop is fine for simple examples like this. But I am currently maintaining a code base that has LOOPs with dozens and dozens -- sometimes a few hundred -- clauses. A single LOOP can extend over multiple pages. It's a nightmare.

Note that LOOP can fail even for simple examples. Suppose I have a list of lists of numbers and I want to collect all the prime numbers. With WITH-COLLECTOR I can do this:

    (with-collector collect
      (dolist (l1 l)
        (dolist (n l1)
          (if (primep n) (collect n)))))

But with LOOP I can't because there is no way for an inner loop to collect into a collector bound in an outer loop. I have to collect the individual sub-lists and then append them, or something like that, which is both inelegant and inefficient.

And if I have a tree of items which I want to walk over and collect all of the once satisfying a predicate, LOOP just doesn't handle that at all. But by separating collection from iteration it becomes trivial:

    (with-collector collect
      (do-tree (item tree)
        (if (predicate item) (collect item))))

Neither WITH-COLLECTOR and DO-TREE are part of CL, of course, but writing them is an elementary exercise (and both are part of ergolib if you really don't want to be bothered).

> LOOP is not extensible

The standardized LOOP is not extensible. Each LOOP implementation is extensible. A common extension is to provide new loop iteration paths.

LispWorks:

  CL-USER 7 > (lispworks:defloop (s-element s-elements) elt length sequence t)
  (S-ELEMENT S-ELEMENTS)

Now we can iterate over sequences in a not performance optimized version:

  CL-USER 8 > (loop for e being each s-element of "hello-world" count e)
  11

  CL-USER 9 > (loop for e being each s-element of '(hello world) count e)
  2

There are portable extensible LOOP implementations. This is then just as non-standard as any self-defined iteration library.

https://research.gold.ac.uk/id/eprint/2344/1/sequences-20070... gives an example to extend SBCL's LOOP (which is based on the MIT LOOP implementation).

There are lots of things in the Common Lisp standard, which are not extensible, but where implementations and libraries provide extensible versions. Example: the CLOS MOP, sequences, hash tables, ...

> Each LOOP implementation is extensible.

Allegro isn't. Neither is SBCL. Nor CCL. Nor CLisp. In fact, the only implementation that offers this feature AFAICT is Lispworks.

> There are lots of things in the Common Lisp standard, which are not extensible

Sure. So? Just because loop is not the only thing in CL that sucks does not make it suck any less.

Pretty much all of those are extensible, especially given that a) they are provided in source code and b) those LOOP macro implementations have internal extension features. The paper I've linked gives an example how to extend LOOP in SBCL for iteration over sequences.

ITERATE is nice, except for one very serious blemish: the accumulation into variables requires a code walker, because the variables are not declared at the top of the iterate form. Macros that require code walkers do not play nicely with other kids. Macrolet/symbol-macrolet, for example, can fail inside interate.

Couldn't you make a similar argument about DO?

I don't see a lot of value in "X is just Y and Z, why use X?" arguments. Clearly it has enough ergonomic benefit that people like using it.

I definitely think the design of the DSL is a little chaotic, not to mention difficult to remember, and having two entirely different DSLs implemented by the same macro is downright ridiculous. But it's definitely good enough at its job and better than not having it at all. If it didn't exist, people would probably be DIYing their own (even worse) versions of it, without the assurance of it being part of the specification and ostensibly a well-tested part of any implementation.

It's still lisp right? You can just do your "(out ...)" like the 2003 guy suggests and not use the format form if you don't like, why get rid of it for everyone else?

And sharing lisp code, is that a thing people do? (this half an insult half a joke) So you can easily enforce your own ideals in your own project.

> why get rid of it for everyone else?

I'm not advocating getting rid of format, or even loop (which is by far the greater of the two evils). I'm pointing them out as cautionary tales for future DSL designers.

Reply on news.ycombinator.com