The Nature of Lisp (2006)

There's a "lisp parens lol" thing among programmers that's so intellectually lazy. I once interviewed someone who I noticed had explicitly mentioned Lisp on their resume, and when as an aside I asked what their thoughts on Lisp development were, all they could muster was "Oh god the parens hyuck hyuck".

I think most people have an irrational dislike of s-expressions, but that at the same time, s-expressions are not as good as their proponents think they are. There's a certain aesthetic "purity" to them that's seductive, but from a practical standpoint, it is quite reasonable to adapt syntax to common patterns. The issue with mainstream languages isn't that they do that, it's that they do it too much.

But there are syntactic schemes besides s-expressions that allow for good flexibility. For instance, in my programming language Earl Grey[1], I use this very simple sugar: `a b: c` <=> `a(b, c)`. So you can use this todo syntax if you want, and it will just work:

    todo "housework":
       item priority(high): "Clean the house."
       item priority(medium): "Wash the dishes."
       item priority(medium): "Buy more soap."

Interestingly this has one advantage over s-expressions: whereas Lisp editors usually need to specify a special highlighting/indent policy for keywords like `let`, here editors can just highlight/indent `a b: c` indiscriminately.

I also think a limited number of infix operators are worth defining in most languages, and I don't mean arithmetic operators (although I'm okay with them). I mean ubiquitous or easily repurposed operators like assignment/declaration, lambda, pairing, or commas. In my view, Lisp-like languages hold to standards of purity that are unreasonable (don't get me wrong, they are quite usable, just not optimal according to any useful metric).

[1]: http://breuleux.github.io/earl-grey/

I really like the Clojure's minimalistic EDN format syntax. In comparison, Scheme and Common Lisp seem like they have superfluous parens.

Generally, you just don't expose your Lisp evaluator to untrusted users. Lisp macros are a powerful language feature, and to allow your user to evaluate arbitrary code is exactly as dangerous as with any other system.

Emacs is not meant to be secure against user scripts. You could even say that's an integral part of its whole philosophy: the user can do whatever she likes, and her scripts too.

It does have some rudimentary safety. For example, color themes by default can't run Lisp code. They are written as Lisp data structures, but they aren't run through the evaluator unless the user explicitly allows it.

Just like every other language.

You either

- Expose the entire language and then there's not much you can do to prevent something harmful.

- Expose a subset of the language perhaps through parsing for harmful patterns or by somehow making potentially harmful functions unreachable.

- Expose a more restricted language (scripting like c/lua) that you control and know its full capabilities.

The only difference I can think of is that it's easier to write flexible and dynamically modifiable software in lisp than in most mainstream languages.

It's up to the programmer to decide what code/data is trusted and untrusted. So you wouldn't `eval` any S-expressions that could be submitted by a user. It's really no different in terms of security implications than any other dynamic language like Ruby, Python, Perl, PHP, Javascript...

AFAIUI you write restricted evaluators to deal with external code. You can dismiss or rewrite any form you want, avoid IO, system calls and such. I never looked into it so maybe it's possible to escape that, it's been used on an irc bot for #clojure, so it's not stupid; but there's always a possibility for crazy ideas like crafting memory patterns and trying to DDoS the evaluator in order for these to become code (complete speculation based on Return Oriented Programming).

Don't use read or eval willy nilly. If you have to use read, at the very least turn off read-eval. I give up on trying to get an asterisk on each side of that variable. No form of escaping seems viable.

What does code-is-a-data have to do with security?!? Do not confuse an exposed `eval` with a compile-time homoiconicity.

Eventually, some of your code will make a call to IO: reading from or writing to disk, responding to a web query, updating a DB, drawing something on the screen, etc, etc.

For example, you could put up a web server written entirely in Scheme or some other dialect. This would process and respond to queries entirely in Scheme, including some sort of engine for generating and serving html. Hacker News itself is an example of this: it's written in a lisp dialect called Arc.

Incidentally, Haskell is functional, but it isn't a lisp.

It's the difference between introducing new nouns and verbs, and introducing new grammar.

With new classes, I add new nouns (Circle, for example). With functions, I add new verbs (move, for example). And I can add adjectives and adverbs (attributes and arguments). This lets me build a "new language" (for handling shapes, in my example).

But macros let you go much further. They let you change the syntax, so that instead of saying "shape.move(newCoords)", you can say "move shape newCoords", for example, even though that was not legal syntax in the language you're writing in. That's where it really becomes a new language - not just new words.

But if you've got "good enough" syntax in the language you're writing in, you might not see that as particularly useful. That is, the less impoverished your out-of-the-box implementation language syntax is, the less you need macros. (Most of the examples say "it's awkward to write this; I can write a macro to make it easier". Well, that's great and all, but awkwardness of writing something that I don't write very often in the first place does not make me want to run out and adopt a macro language.)

My personal favorite is the series library. I wrote a bit about it here: https://news.ycombinator.com/item?id=9078155

Statically compiled regular expressions. Efficient pattern matching implementations. Generating a boilerplate for an ORM. Embedding languages with arbitrarily complex syntax. There are hundreds of things that you simply cannot do with functions but are trivial with macros.

...shades of Spring application.xml

Do you have concrete examples of this difficulty?

I very often find it difficult to understand code in non-Lisp languages, too. For example, you may think you know Ruby, but that has nothing to do with understanding Rails or its myriad metalinguistic tricks.

Abstractions are often difficult to understand, and require explanations. Introducing a new abstraction, regardless of implementation language, puts the burden on you to define it with clarity, to document its purpose and meaning, and so on.

Lisp macros let you define abstractions in a certain way that's quite powerful. Fortunately, Lisp systems also tend to have good provisions for online documentation, and make available macro-expansion facilities that immediately explain exactly what a given macro is doing.

In general, the more you can package up into single abstractions, the more someone will have to learn to thoroughly understand your code base. If those abstractions fit the problem domain well, hopefully they will ease the overall cognitive load of working in the system, even somewhat early on, but there will always be some ramp-up time. I'm not at all sure it's a big problem, just something you need to be aware of.

Experience says no.

There are several reasons, but

    * macros are only used when functions doesn't make sense
    * there are documentation
    * most follow the conventions of the language

If macros are used properly, i.e., for implementing nice, multi-staged, composable eDSLs, instead of just another anaphoric if or similarly useless "extension", then a Lisp codebase is much easier to read than anything that does not allow implementing macro-based eDSLs.