The syntax is just off-putting and I was never compelled to learn it in school, so I remain in algol-land. If time and focus allowed I'd do the SICP self study path but it's too easy to surf HN and Reddit instead.
Algol languages on the other hand, and math even on paper, are a rather wild bunch: they mix infix operators like plus/minus (1+1), multiplication/division (2*2), have postfix operators like faculty (9!) and even user defined functions f(x, y) in prefix notation. That should be way more off-putting theoretically, but people just grew up with this "mess" and think that fine.
Disclaimer: besides Lisp, I program in Perl or Shell too ;-)
You can write down the syntax of a call to a macro which has not yet been written, and Lisp will read it for you, giving you the tree. It just won't be able to expand it since the semantic definition through the macro name isn't there.
A conventional language will not do that. C++11 will not correctly read the shape of a C++17 phrase structure, and just complain about it not being defined.
Lisp macro can treat their arguments as a flat stream of tokens and parse them, even with LALR(1) parser generator or whatever. It's not not the typical way most macros are written. Macros leverage the ability of the invocation syntax to express pre-parsed nested structures.
Even if a macro goes crazy like that, it's still self-contained; the churning toilet bowl of parsing it has whipped up is cleanly confined within the boundaries of the macro call, which can be cleanly substituted anywhere an expression can be used.
(let (x 5) (* x x))
(let ((x 5)) (* x x))
Let me ask you a question: Are these two expressions equivalent?
(f (g x))
(let ((y (g x))) (f y))
CL has a core language with a simple syntax. It also has a lot of macros in its standard library.
Because it parses, I don't have to correct the author's punctuation, just their understanding of the necessary shape . The wrong shape they wrote has a coherent syntax, and we can discuss the problem in terms of differences in syntax, rather than differences between gibberish and valid syntax.
This is why your question is able to be, "are these expressions equivalent" and not "are these two token sequences equivalent".
I can write a TDD test case for a new macro that doesn't exist yet, which won't bring down the program.
E.g. (test (my:let (x 5) (+ x x)) 10)
This will catch the error that foo is not defined and report that the test case failed instead of returning the expected 42 value. The next test case in the same file can execute.
You couldn't have a test for new C++ or Java syntax, in a C++ or Java file being processed by a compiler that wasn't maintained in order to recognize the syntax. The file will blow up at parse time, not allowing any of the test cases in it to proceed.
> You need to know the syntax of the f macro to answer the question.
But I can see what that syntax is: it's (f (g x)) right there: i.e.
/\
f /\
/\ ()
g /\
x ()
I'm not struggling with syntax, though, just connecting it with a meaning.
That's not the definition of syntax.
If you're implementing a compiler, then you might use "syntax" as a shorthand for those parts of the syntax that are processed in a particular "syntax checking" stage of the compiler. But that's not all it is.
> From looking at the use, I don't know whether it is valid shape for f
That, on the other hand, is close to a definition of syntax: The rules for what has a valid shape and what doesn't.
The Lisp reader just reads s-expressions. It knows nothing about Lisp syntax and thus does not check it.
It was the reader which settled all the syntactic facts about what operation is applied to what operands.
At this point we could move the goalposts: even eval and compile don't know anything about Lisp syntax. Because, look: (- 10) and (- 10 1) behave differently: the 10 operand changes semantic role between minuend and subtrahend based on syntax. All that eval knows is that it's passing arguments.
A function can treat its argument values as the tokens of a syntax which are shored up according to arbitrary phrase structure rules, similarly to Unix programs like find and tcpdump.
The reader does not know anything about operands or operators. It does not even know if the thing read is data or code. It also does not know if it is valid code. Enter (3 4 *) and the reader happily takes that expression, not caring about operators and operands. The reader will print zero warning about such expressions.
To really know what the symbols in an expression actually are, one would need eval, compile or a code walker. Are they used as literal symbols? variables? functions? macros? block names? type names? ...
> Because, look: (- 10) and (- 10 1) behave differently: the 10 operand changes semantic role between minuend and subtrahend based on syntax. All that eval knows is that it's passing arguments.
That's semantics, not syntax. EVAL and COMPILE check that it is a function call, what arglist it expects (zero or more args, &optional args, &rest args, keyword args, and process them recursively. But Lisp has also special operators, macro operators and a lambda expression. There the syntax requirements are different or even context depended.
Also if Lisp reads the expression (foo * +), the reader does not know if FOO is a function or a macro, it thus does not know what * and + are. Are they variables? Are they operators? Something else? They don't even need to have a definition at read time.
The reader doesn't know about opperands or operators, but it certainly knows a symbol from an integer or string.
We've cleverly arranged things so that it doesn't have to know anything else in order to produce output that, in some cases, can be evaluated directly.
> That's semantics, not syntax
Syntax versus semantics is sometimes about where you put the goalposts.
What if I make it a macro that looks at the argument count and chooses a different function?
(defmacro minus (&rest args)
(if (eql 1 (length args))
`(unary-minus ,(car args)) ;; takes exactly one arg
`(nary-minus ,@args))) ;; blows up if not given at least two args
Everything is syntax! Adding two floating-point value is syntax: the representation has to be parsed into sign bit, exponent and mantissa, and so on before anything happens.
Sure, those fields are accessed conveniently by position, but so are the elements of the list (sign exponent mantissa), and our goalposts say that that is syntax.
The parsing never stops. The CPU fetches instructions, decodes (== parsing!) and executes.
When the Lisp run-time looks at a value to extract the tag bits, and then further delves into the object accordingly, it is parsing syntax.
That's the s-expression syntax. Beyond that, symbols have no special meaning for the reader.
> Everything is syntax
Syntax describes the structure of a language.
Wikipedia: "the syntax of a computer language is the rules that defines the combinations of symbols that are considered to be correctly structured statements or expressions in that language."
CLHS 1.4.1.2 "This specification uses an extended Backus Normal Form (BNF) to describe the syntax of Common Lisp macro forms and special forms."
So the ANSI CL standard claims that its macro forms and special forms have syntax and that this syntax is described by an eBNF?
Example: DO
do ({var | (var [init-form [step-form]])}*)
(end-test-form result-form*)
declaration*
{tag | statement}*
* (do "hello")
debugger invoked on a SB-KERNEL::ARG-COUNT-ERROR in thread
#<THREAD "main thread" RUNNING {1004AC01C3}>:
Error while parsing arguments to DEFMACRO DO:
too few elements in
("hello")
to satisfy lambda list
(VARLIST ENDLIST &BODY BODY):
at least 2 expected, but got 1
Woops? There is a parser? There is an error detected?
So, beyond the the reader SBCL seems to claim it has a parser?
But, above, the message "error while parsing arguments to DEFMACRO DO" is just a destructuring error on "macro lambda list" pattern match on a piece of datum: the unevaluated macro call. That pattern matcher has no idea what DO means, in much same way that the reader doesn't know what DO means.
The reader has done so much work, that the remaining parsing is just a very easy job which enjoys easy positional access to everything in the tree, at every level, at worst having to loop over a sequence of similar items.
The syntax notation like:
do ({var | (var [init-form [step-form]])}*)
(end-test-form result-form*)
declaration*
{tag | statement}*
The init-form can be anything: a struct literal, vector, string, character. Yet, I don't see the rules for that in this grammar; they given elsewhere, and just assumed here.
If everything were to be made explicit here, the way a handful of parentheses have been made explicit, it would be pretty large specification, of which the do-specific parts above would be small.
The do-specific material is just a handful of self-contained patterns applied to individual objects that have already been parsed out by the reader.
For instance, the pattern pattern {var | (var [init-form [step-form]])}* is understood to be applied to that part of the form which can be conveniently retrieved as (cadr form). The amount of work left is very small, of low complexity. The pattern is not a recursive language; it is a regular language (if we pretend that var, init-form and step-form are terminal symbols).
When the (var init-form step-form) is matched against the corresponding object, the step-form is easily retrieved by position, regardless of the complexity of init-form. This is not true of the unparsed expression in which scanning across an init-form to get to step-form is a complex step. The reader has done the work which allows this stage of processing to treat "init-form" as if it were a terminal symbol in a simple regular grammar that is just concerned with a positionally-determined portion of the do syntax.
It looks like Lisp has a lot of syntax beyond the read syntax if we look at all of the forms together. They all share the read syntax, but have their unique bits that drive their shape. The more of them there are, the larger is the proportion of the language syntax that is contributed by those unique bits. Each time we write a macro and document it, the read syntax doesn't grow, but the amount of deep syntax does.
From the perspective of any one S-expression, though, the bulk of the specification of its syntax is in the domain of the reader. The pattern match for a specific shape like do is a small amount of easily digestible additional information.
'means' is in the domain of semantics. We talk about syntax: structure.
But what we know at this stage: DO is the CL:DO macro. So we know the actual operator, not just the symbol. We also know that it is a macro. We also know that the macro has a macro lambda list (CLHS 3.4.4 describes the syntax of those). We can now check the macro form against the actual CL:DO macro lambda list structure. And so on...
Thus we are now talking about the Lisp language, its operators and its structure requirements.
That's the purpose of a language parser: it has to know which symbols/identifiers are actual elements of the language, which operators these actually are and retrieves the syntactic constraints/rules for those.
In a language like Common Lisp (or similar dialects) this is 'difficult', since there are lexical bindings, compile-time side-effects and even lexical bindings to establish new syntax (macrolet).
> The syntax notation like: ... is relying on read syntax; it has parentheses in it!
It describes constraints on the structure (!) of valid Lisp forms. -> syntax!
(do () () "foo") -> Ill-formed DO end test list
(do () (t) "foo") -> no legal go tag
(do ((a b c d)) (t)) -> illegal form for a DO varlist
None of these things are caught by the READER. All of these are actual warnings from SBCL.
> The init-form can be anything: a struct literal, vector, string, character. Yet, I don't see the rules for that in this grammar; they given elsewhere, and just assumed here.
The rules are in the language standard. The non-terminal INIT-FORM is specified to be a FORM. Which then is defined elsewhere.
> The init-form can be anything
Actually it can't. It must be a form (see the CLHS syntax description for DO). Thus ((foo)) for example is not valid, since it is not a valid Common Lisp form.
> From the perspective of any one S-expression, though, the bulk of the specification of its syntax is in the domain of the reader.
This is tautologic.
I was claiming that Lisp has structural constraints/rules on top (!) of s-expressions (and especially s-expressions as data) and gave the example of the Common Lisp standard, which describes these in an eBNF syntax.
Source code is actually simple nested list data. For example the following snippet, which binds two variables A and B, defines a local function and then calls this local function. This is just an s-expression, which consists of symbols, numbers and lists.
Even though s-expressions have a syntax with numbers, parentheses, symbols, ... - they can not only be written in a text editor, one can also directly compute with them using the typical Lisp operators for lists, numbers, strings, symbols, ...
The following enters literal data, which is not executed as code.
CL-USER 5 > '(let ((a (+ 1 2))
(b (* 2 3)))
(flet ((add3 (a b)
(+ 3 a b)))
(add3 a b)))
(LET ((A (+ 1 2))
(B (* 2 3)))
(FLET ((ADD3 (A B)
(+ 3 A B)))
(ADD3 A B)))
CL-USER 6 > (subst 'expt '* *)
(LET ((A (+ 1 2))
(B (EXPT 2 3)))
(FLET ((ADD3 (A B)
(+ 3 A B)))
(ADD3 A B)))
CL-USER 7 > (eval *)
14
LET is a syntactic form and expects a list of (variable value) pairs as its first sub-expression. Then follows a body of subexpressions, where the variables can be used.
FLET is a another syntactic form: it expects a list of function definitions and a body. The function definition starts with a name, followed by an arglist and then by a body.
In reality this is more complex, since LET and FLET allow declarations, functions allow complex argument lists, declarations, documentations, etc.
So for FLET the top-level EBNF syntax would be like:
flet ((function-name lambda-list
[[local-declaration* | local-documentation]]
local-form*)*)
declaration*
form*
well->people[2].are(likely, to, like, this, better);
((Person->are ((array-at (Prefix->people well) 2)) ....
(Yes, this is an exaggerated example but this really is the point. Also, static types. If there's anything of overarching regularity in what you type, it better be syntax or it will be hard to check and take advantage of. This is what Algol-style languages do for systems programming.)
(peopler-simplifier well people 2 are likely to like this better)
This to me is actually the crisis for lisp. Any codebase requires you to learn all the various macros that do:
peopler-simplifier == $1->$2[$3].$4($5, $6 $7 $8 $9);What is being discussed in this sub-thread is an oft-cited criticism of Lisp's "power" being too much for its own good, but rarely (if ever) a reflection of idiomatic practice.
Too much power for its own good is one thing, but being just not optimized for the common thing is another. How would the following function, randomly picked from what I have open on github right now, look in "Common" Lisp? And if I ask 3 persons to do it, what is the likelyhood that they would pick more-or-less the same approach?
Edit_Array buffer_batch_array_from_linked_list(Arena *arena, Batch_Edit *batch, i32 count){
Edit_Array result = {};
result.count = count;
result.vals = push_array(arena, Edit, count);
i32 counter = 0;
for (Batch_Edit *node = batch;
counter < count && node != 0;
node = node->next){
result.vals[counter] = node->edit;
counter += 1;
}
return(result);
}It just uses names operators in prefix position. foo += 1 then is (incf foo). for (...) { ... } is then (loop for ... do ...). ...i32 counter = 0;... is then (let ((counter 0)) (declare (type i32 counter)) ...)
and so on...
(defun buffer-batch-array-from-linked-list (arena batch count)
(let ((result (make-edit-array
:count count
:vals (push-array arena edit count))))
(loop :for counter :below count
:for node := batch :then (next node)
:for vals := (vals result)
:until (null node)
:do (setf (aref vals counter)
(edit node)))
result))
Could you adorn this code with type declarations? Yes, and you'd get speed and/or safety benefits by doing so. Depends how much this function "matters" in the larger context.
But in your lisp version "next" and "edit" appear to be global methods. That seems like it could lead to a lot of naming collisions. Could you still have a document type that you can call "(edit document patch)" on? A preliminary Google search suggests that CLOS methods can't have definitions with a different number of parameters, so now you're stuck with a global "edit" method that can only ever take one parameter.
(defclass a ()
((value :reader value)
(next :reader next)))
(defclass b ()
((value :reader value)
(next :reader next)))
But, if generic functions aren't your thing, you can skip calling the functions even and just do
(slot-value x 'next)
x->next
Regardless of all of that, symbols themselves belong to a namespace, called the "package". So your "edit" and my "edit" methods can have a different number of arguments without issue. (If they do have different arguments, they must mean different things.)
E.g. you might have some doc-model:edit symbol and file-utils:edit or whatever. The symbols have the same name, but are different objects in different packages.
The generic function doc-model:edit is therefore unrelated to file:edit.
Packaging at the symbol level means that the package system is relevant to anything whatsoever that is named by a symbol. If you have some logic programming system where you declare some named rules or facts, they can easily go into a package. Symbols which are quoted individually or in a larger literal are in the package system all the same; it's an independent layer.
(replace (make-array count) batch) Edit_Array buffer_batch_array_from_linked_list(Arena *arena, Batch_Edit *batch, i32 count)
{
Edit_Array result(arena, count); // Edit_Array has this thingy called a constructor
foreach (node in batch) // Batch_Edit * is somewhere/somehow declared iterable
{
if (result.full()) // Edit_Array has internal fill index, checked against count
break;
result.add(batch->edit); // Edit_Array has method to add and increment fill index.
}
return result; // not a function, drop the parens
}
After a while I realized it - I may have programmed in Common Lisp but never professionally... on a professional code base. Only ever on open source thingies, and small side projects. Paid or unpaid.
I've never seen industrial common lisp code. My last job had some but I never got to work on that project, and no longer can see it :(
Is there an example somewhere of this? C++ examples would be things like Chrome, C would be lots of open source OS's and kernel drivers. Is there something equivalent for CL?
SICP was first published in 1984 [2]
[1] https://stackoverflow.com/a/69261645/3414663
[2] https://en.wikipedia.org/wiki/Structure_and_Interpretation_o...
Philip Wadler's paper discussed and linked from here is also interesting: https://www.wisdomandwonder.com/link/1055/why-calculating-is...
For example SICP uses a so-called special form called cons-stream, which is in no Scheme definition (it was in no Scheme reports) and for which no implementation is given in the book, since it can't be implemented by the Scheme subset used for the book. It assumes that it is somehow provided without giving any details.
I recently made a comment on this topic here:
https://news.ycombinator.com/item?id=33049170
There are probably some engineers who have "dyslispia", to whom the normal example with the usual parentheses doesn't look any better than the messed up examples.