Homoiconicity Revisited

Programming languages never live or die on their technical merits. Javascript is a hot mess, and yet it is one of the most popular languages in the world.

Also:

> macros become much harder once you move away from leading parenthesis

That is not really true. You do lose the backquote syntax, but that is only useful for writing the simplest most trivial macros. That turns out to be the kind what most people write and so backquote is a big deal, but if you want to write more complex macros (e.g. cl-who) then backquote becomes less useful and the surface syntax becomes less relevant.

This is... an extraordinary claim. Prolog uses foo(hello, world) syntax (note the comma) and has no problem with macros or any other manipulation of such terms. Note that there are longer, better reasoned, and higher voted answers to that StackOverflow question than the accepted answer which opts for "macros would become more difficult" FUD.

Languages such as Python and C draw clear distinction between literal values on one hand and flow control statements and operators on the other. Numbers, strings, arrays, structs are first-class data. Commands, conditionals, math operators, etc are not; you cannot instantiate them, you cannot manipulate them.

What homoiconic languages do is get rid of that (artificial) distinction.

Lisp takes one approach, which is to describe commands using an existing data structure (list). This overloading means a Lisp program is context-sensitive: evaluate it one way, and you get a nested data structure; evaluate it another, you get behaviors expressed. The former representation, of course, is what Lisp macros manipulate, transforming one set of commands into another.

Programming in Algol-descended languages, we tend to think algorithmically: a sequence of instructions to be performed, one after the other, in order of appearance. Whereas Lisp-like languages tend to encourage more compositional thinking: composing existing behaviors to form new behaviors; in Lisp’s case, by literally composing lists.

Another (novel?) approach to homoiconicity is to make commands themselves a first-class datatype within the language. A programming language does not need swathes of Python/C-style operators and statements to be fully featured; only commands are actually required.

I did this in my kiwi language: a command is written natively as `foo (arg1, arg2)`, which is represented under the hood as a value of type Command, which is itself composed of a Name, a List of zero or more arguments, and a Scope (lexical binding). You can create a command, you can store it and pass it around, and you can evaluate it by retrieving it from storage within a command evaluation (“Run”) context:

    R> store value (foo, show value (“Hello, ”{$input}“!”))
    R> 
    R> input (“Bob”)
    #  “Bob”
    R> 
    R> {$foo}
    Hello, Bob!

..

When it comes to homoiconicity, Lisp actually “cheats” a bit. Because it eagerly (“dumbly”) evaluates argument lists, some commands such as conditionals and lambdas end up being implemented as special forms. They might look the same as every other command but their non-standard behaviors are custom-wired into the runtime. (TBH, Lisp is not that good a Lisp.)

Kiwi, like John Shutt’s Kernel, eliminates the need for special forms entirely by one additional change: decoupling command evaluation from argument evaluation. Commands capture their argument lists unevaluated, thunked with their original scope, leaving each argument to be evaluated by the receiving handler as/when/only if necessary. Thus `AND`/`OR`, `if…else…`, `repeat…`, and other “short-circuiting” operators and statements in Python and C are, in kiwi, just ordinary commands.

What’s striking is how much non-essential complexity these two fundamental design choices eliminate from the language’s semantics, as well as from the subsequent implementation. kiwi has just two built-in behaviors: tag substitution and command evaluation. The core language implementation is tiny; maybe 3000LOC for six standard data types, environment, and evaluator. All other behaviors are provided by external handler libraries: even “basics” like math, flow control, storing values, and defining handlers of your own. Had I’d tried to build a Python-like language, I’d still be writing it 10 years on.

There are other advantages too. K&R spends chapters discussing its various operators and flow control statements; and that’s even before it gets to its stdlibs. I once did a book on a Python-like language; hundreds of pages just to cover the built-in behaviors: murder for me, and probably not much better on readers.

In kiwi, the core documentation covering the built-in data types and how to use them, is less than three dozen pages. You can read it all in half an hour. Command handlers are documented separately, each as its own standardized “manpage” (currently auto-generated in CLI and HTML formats), complete with automated indexing and categorization, TOC and search engine. You can look up any language feature if/when/as you need it, either statically or in an interactive shell. Far quicker than spelunking the Python/C docs. A lot nicer than Bash.

Oh, and because all behaviors are library-defined, kiwi can be used as a data-only language a-la JSON just by running a kiwi interpreter without any libraries loaded. Contrast that with JavaScript’s notorious `eval(jsonString)`. It wasn’t created with this use-case in mind either; it just shook out of its design as a nice free bonus. We ended up using it as our preferred data interchange format for external data sources.

Honestly, I didn’t even plumb half the capabilities the language has. (Meta-programming, GUI form auto-generation, IPC-distributable job descriptions…)

..

Mind, kiwi’s a highly specialized DSL and its pure command syntax makes for some awkward reading code when it comes to tasks such as math. For instance, having to write `input (2), + (2)` rather than the much more familiar `2 + 2`, or even `(+ 2 2)`. Alas it’s also proprietary, which is why I can’t link it directly; I use it here because it’s the homoiconic language I’m most familiar with, and because it demonstrates that even a relative dumbass like me can easily implement a sophisticated working language just by eliminating all the syntactic and semantic complexity that other languages put in for no better reason than “that’s how other languages do it”.

More recently, I’ve been working on a general-purpose language that keeps the same underlying “everything is a command” homoiconicity while also allowing commands to be “skinned” with library-defined operator syntax to aid readability. (i.e. Algebraic syntax is the original DSL!) It’s very much a work in progress and may or may not achieve its design goals, but you can get some idea of how it looks here:

https://github.com/hhas/iris-script/blob/f9d9298824d05eccb22...

Partly inspired by Dylan, a Lisp designed to be skinnable with an extensible Pascal-like syntax, and also worth a look for those less familiar with non-Algol languages:

http://www.gwydiondylan.org/books/drm/drm_7.html

And, of course, by Papert’s Logo:

https://www.amazon.com/Mindstorms-Children-Computers-Powerfu...

There is no need for ugly, cumbersome syntax objects that destroy Lisp.

Whether or not the recording of source loc info is enabled is controlled by a special variable. It is disabled by default, because you probably don't want that overhead if you're calling the reader for reams of data. Functions like compile-file and load enable it locally.

To take advantage of the info for error reporting in a user defined macro, you can simply do this:

   (defmacro mymacro (:form f arg1 arg2 ...) ;; get the form using :form
     (when (somehow-no-good arg1) ;; something wrong with arguments
       (compile-error f "first argument ~s is no good!" arg1))
     ...)

   (foo.lisp:10) mymacro: first argument 42 is no good!

EDIT: If anything, I’d define homoiconicity by the ability to write the same program (as EVAL sees it) in a variety of different textual formats: e.g. ‘foo is equivalent to (quote foo) and both programs are indistinguishable to EVAL

Why is homoiconicity "very very important"? Nobody seems able to say.

> You have people around the internet with very little programming experience discussing all day about concepts they hardly understand, because they have not programmed a lot to really grasp it.

The people I quoted were all principal developers of arguably the most successful Lisp derivative in the past few decades. The annual Scheme Workshop is almost entirely Racket these days. I doubt you are in a place to claim that "they have not programmed a lot to really grasp [homoiconicity]". So I'm confused what your point is here.

But what is the impossible idea? What is homoiconicity? Can you give a succinct definition that is consistent with your usage?

I don't meant this to sound so antagonistic, but I've seen multiple times where people try to argue in favor of it and come up short when pressed for details.

Unfortunately, I do not think I am sufficiently skilled to carry the debate forward. I really recommend looking through Shriram's Twitter exchange to see some of his rebuttals on the topic.

So similarly as you state about homo-iconicity some languages are simply purer than others.

Mm what I meant to get at was that you give a definition of "meaning", which is that "a word means what the people who use it mean by it", and your use of the phrase "let's say" makes it sound like this is a charitable admission on your part. And my point is that this is what it means for a word to "mean" something — it's entirely based on what people who use it mean by it, so the way you say "let's say" was rubbing me the wrong way, so to speak.

> I actually agree... the article (and the previous one) are mostly consequences of starting out in the belief that homoiconicity is something to strive for, and then failing to understand what homoiconicity would really be -- but discovering the things to strive for in the process.

I did not take the time to read the previous article, but I regret that now. I will go back and read it. :)

I don't think I did a good enough job saying that I think this is among the best articles on homoiconicity that I've read! I meant for my lament to be more general and not specifically directed at you, but more just at those people striving for homoiconicity without a clear idea of what that means. I think, rereading parts of this article again now, we're more aligned than I had previously though. I'm sorry for not realizing that earlier!

Words only have meaning due to community consensus. Yes, we have dictionaries to record what the consensus is (or, more accurately, what the consensus was at a particular point in time), but the dictionary cannot predict semantic drift, nor can it adapt with sufficient speed in the modern era of light-speed communication.

The dictionary only works because most people who use a given word already use it consistently with how the dictionary describes it. It is not the case that most people learn the word "dog" from the dictionary; the dictionary's definition is "correct" because it aligns with the way people already use the word. If Merriam-Webster published a revised definition stating that a dog is "a large, four-legged creature with wings which breathes fire", people would immediately agree that the dictionary is wrong. It only reflects words as they are used, not words as they must be used. (The Merriam-Webster official Twitter account is adamant in promoting this view of dictionaries, for what it's worth.)

Emailed, still shadow banned, should I email again?

Thanks, done