The Problem with Lisp
heinrichhartmann.com
heinrichhartmann.com
The evaluation order in Lisp is not "highly unpredictable", it is dependent on the form.
I don't understand why he even constructs this strawman of "you must start in the innermost sub-form".
That's not a rule that anyone ever made. That's the rule for arithmetic expressions, sure, but what does that have to do with Lisp?
The rule in Lisp – as he later admits – is "look at the first symbol in the (outer) form".
And the whole point of special forms is to make your own rules about evaluations. In order to help the programmer.
Can you obfuscate the meaning? Hell, yes! Is that a reason not to allow it and to force obfuscated code by using unsuitable evaluation orders everywhere? Hell, no!
And even in this bad example Lisp is certainly not more unpredictable than any other language.
Let's take C:
int func(int a, int b)
{
return a + b;
}
I think the author has misled himself into some extreme notion of the divine simplicity and purity of Lisp that nobody ever claimed to exist.
Let me first comment on your C example:
> "a + b; // Oh gosh, what is the + operator?"
There is no problem with the "+" operator. It's a language builtin. You can look that up in the manual.
In the Lisp case, I did not pick on the "+" operator, at all. The problem is a line below:
> "(sum x) ;; ??? Where does "sum" come from?"
There is no function "sum" at all! It just looks like a function call, which is it not. You have to know what the enclosing special form "let" does to it's arguments.
You could have those problems with C macros as well, but not in this example.
> Can you obfuscate the meaning? Hell, yes!
> Is that a reason not to allow it and to force obfuscated code by using unsuitable evaluation orders everywhere? Hell, no!
I am not picking sides here. I am pinpointing a source of confusion. The Python/Ruby authors took another path and made the language less powerful, so they don't give developers the power to obfuscate the code in this way.
Most languages have constructs which don't behave like function calls and which have special evaluation rules. A simple example is the IF statement. It takes a condition and two expressions. Depending on the condition only one of the expressions will be evaluated. If you look around, you'll find a bunch of such constructs in most languages. Lisp also has such an IF operator. Lisp additionally allows us to write any amount of additional operators which have evaluation rules, which are not the evalation rule of a function call.
Is the IF statement irregular? It isn't. It's just one of many operators which does not follow the evaluation rule of a function call.
Something like Ruby has a lot of weirdness in its calling mechanism. Python has strange non-local transfers going on. There are lots of ways we can perceive 'irregularities'.
Well, Python and Ruby have an "eval" statement. So "code as data" is supported, but discouraged. In contrast Lisp macros / "code as data" are commonly used.
Why is there a difference in culture? Is the only reason, that strings are a lot harder to manipulate than lists? Is there more to it? Maybe translated code is harder to maintain and reason about? Maybe debugging and profiling is a lot harder (this is true for C macros at least).
I have just shared one confusion that I was struggling with.
I found python and ruby much easier to read and pick up. One reason for this is, that programmers are not allowed to define new irregularities. Maybe this insight is helpful for others, who are getting Lisp as well. At least this was my motivation for writing this post.
Ruby and Python represent code as text. That's what Lisp does, too. One has a file of code. But Lisp has something which neither Python and Ruby do: the text is actually a data structure: lists, strings, numbers, vectors, symbols. Lisp provides a function READ.
You can read code:
CL-USER 73 > (with-input-from-string (stream "(1 + 2)")
(print (read stream))
(values))
(1 + 2)
This then makes it easy to compute valid code and execute it:
CL-USER 74 > (with-input-from-string (stream "(1 + 2)")
(let ((code (print (read stream))))
(rotatef (first code) (second code))
(print code)
(print (eval code)))
(values))
(1 + 2)
(+ 1 2)
3
CL-USER 75 > (defmacro infix (expression)
(setf expression (copy-tree expression))
(rotatef (first expression) (second expression))
expression)
INFIX
CL-USER 76 > (infix (1 + 3))
4
CL-USER 77 > (macroexpand '(infix (1 + 3)))
(+ 1 3)
T
Or use infix conversion during reading. Write your code with sections of a more conventional syntax:
CL-USER 80 > (defun hypot (a b)
"Compute the length of the hypotenuse of a right triangle
with sides A and B."
#I( sqrt(a^^2 + b^^2) ))
HYPOT
CL-USER 81 > (hypot 1 2)
2.236068
CL-USER 84 > '#I( sqrt(a^^2 + b^^2) )
(SQRT (+ (EXPT A 2) (EXPT B 2)))
If you look at a Lisp interpreter, it actually executes Lisp data as code. Macros transform Lisp data as code. This is far far away from what Ruby or Python do. Python provides some access to its AST - that's also far away from what Lisp does.
Lisp macros don't define irregularities. They do code transformations. Stuff where the language designer/implementor of, say, Python refuses to provide a control structure, you can do it yourself in Lisp in a few minutes or you can spend years to implement really cool stuff. The Common Lisp object system started as a large portable library (with only very little system dependent code), which code be loaded into Lisp and which the provided a very elaborate object system. This also included some clever macro code.
I think titles like 'The problem of Lisp' may need a bit more reflection and patience. What is new for you is old news for the Lisp community. The exact same problem report can be already read in Lisp books from the 1960s. It's a bit like blogging 'The world isn't flat!.;-)
Forgive me for the baity title ;) I realise this is not a new discovery.
> Python and Ruby use flat strings as input to eval. That's an entirely different thing.
I understand this. I was responding to the earlier assertion "Neither Ruby nor Python support the idea of 'code is data'." which is not entirely correct. You can clearly supply code as string data. (Python's namedtuple does make use of this). It's just not nearly as powerful.
Thanks for providing the examples. The infix macro is pretty amazing! I will make use of this #I stuff.
That's just some kind of eval feature. Many languages can execute strings of code at runtime - for example one could write C code to a tmp file, call the compiler, and load the object file -> that's how some Lisp-to-C compilers work. But it's just the usual C code as text.
That's not 'code is data'. Code is data means that code is actually written in a data structure which is serialized and de-serialized. In Lisp code is written as s-expressions.
'code is data' does not mean I can treat code as a string and evaluate it, it means: code is actually read, written and manipulated as structured data - and not as blob of text.
Ruby code is written as textual Ruby code, where Lisp is written as Lisp programs in s-expressions.
What you tell me is the code is text, which can be somehow executed at runtime, which is an entirely different thing.
The mechanisms they give you are more than enough, though. Ruby especially favours this kind of thing a lot; how many people really have any idea what's going on (and how many of them beyond "something to do with method_missing and instance_eval") when they write something like:
FactoryBot.define do
factory :foo do
bar { 'baz' }
end
end
You cannot simply let your eyes randomly land on some (a b c) in Lisp code and assume that a is a function being called with arguments b c. You have to read the whole sentences and phrases, so to speak.
No language is immune to this. Can you assume that * X is a pointer dereference? No, it could be a declarator. Or a fragment of a multiplicative expression. Or the interior of a comment as in /* X.
(a, b) could be a comma expression, or the argument list of a function call.
I can't think of any language in which I can extract any substring of a sentence that could be a syntactic unit on its own, and correctly understand its meaning without knowing the rest of the sentence.
Lisp has to be scanned from the outside in: we have a defun, which encloses a let, and so on. When you're familiar with the piece of code you're working on, you can then take considerable shortcuts. You know you're in the middle of a function, even though the defun is off the screen. You know you're in the middle of the progn-like body of some construct (a loop, or let or whatever), so all the lines of code you see are evaluated forms.
Yes, to some extend.
> Can you assume that * X is a pointer dereference?
[Aside: The C declaration syntax is particularly terrible. In a declaration 'star' means "address of", elsewhere it means "dereference". https://books.google.de/books?id=bXGhBAAAQBAJ&printsec=front...]
> I can't think of any language in which I can extract any substring of a sentence that could be a syntactic unit on its own, and correctly understand its meaning without knowing the rest of the sentence.
Well, most languages have expressions/statements that roughly correspond to source lines. When I read a line of python like this:
self.raw_requestline = self.rfile.readline(65537)
I can be 100% sure that:
- `self` is an object with an attribute rfile
- `=` is the assignment operator
- there is a method `readline` that get's called with an integer argument 65537.
All this without checking any enclosing socpes.
A random line of lisp:
(setcdr current (list :args (cdr current)))
could mean anything depending on the enclosing scopes. Pretty sure setcdr,list,cdr are function calls but who knows. You don't have enough information to decide this from the snippet provided.
I think this is overstating things, eg, the class `self` belongs to might have defined __getattribute__, __setattr__, etc, so the things you know for 100% certain about that line of code in isolation don't really tell you anything about what it does (self does not definitely have that attribute, and that might not be assignment, at least); most of that you assume based on common sense, conventions, figuring the code is "normal", etc, the same as in Lisp. And that's ignoring a trivial case like if the previous line was `f('''`, where f may or may or not call eval.
In some Lisp-like languages such as Scheme, evaluation of function arguments is unspecified, similarly to C.
Lisp is slightly more difficult than some other languages because lists/symbols are both used in code and data, because there are a bunch of different types of lists, and because code is a token tree.
(defun silly (a b c) (let ((my-sum (+ a b c))) (* my-sum (+ 1 my-sum))))
Lisp is written like that:
(defun silly (a b c)
(let ((my-sum (+ a b c)))
(* my-sum (+ 1 my-sum))))
Any definition has this pattern:
(DEFsomething name ...)
Function definitions have this pattern:
(DEFun name arglist body)
(defun name arglist
body
...)
(DEFmethod name arglist
body
...)
LET is:
(LET binding-list
body
...)
((arg0 value0)
(arg1 value1)
...
(arg2 value3))
If you give shortcuts to it:
df function definition
bl binding list
nm name
bd body
al arglist
df nm al
bd
LET bl
bd
prog bl
bd
LAMBDA al
bd
DESTRUCTURING-BIND al
form
bd
BLOCK nm
bd
CATCH nm
bd
Basically much of Lisp consists of a small number of basic patterns which are being combined. lists, assoc lists, property lists, arglists, binding lists, body, ...
You've learned then a new alphabet of visual/structural patterns. Beyond that, code reading gets intuitive. Names give a clue what kind of pattern combination follows.
The IDE can then help with syntax highlighting, syntax documentation display, etc.
Lisp can even show you useful code layout:
CL-USER 68 > (let ((*PRINT-RIGHT-MARGIN* 30))
(pprint '(defun silly (a b c) (let ((my-sum (+ a b c))) (* my-sum (+ 1 my-sum))))))
(DEFUN SILLY (A B C)
(LET ((MY-SUM (+ A B C)))
(* MY-SUM (+ 1 MY-SUM))))Obviously there is lot of community know-how, that I am not aware of. Thanks for explaining them in detail. Learning to spot all those pattern is crucial if you want to be come effective at reading lisp. For the untrained eye lisp code often looks horrid.
I don't think we disagree, though. All I am saying is that: "Riding the Lisp bicycle has been difficult for me. Here one thing I struggled with a lot." Maybe other people have the same difficulty and find this helpful.
You were not knowing that there is actual structure and thus you could not see the forest through the trees.
Lisp looked very good and highly readable; I liked it very much right away and was able to hit the ground running.