> "in SBCL interpreted code is not safer or more debuggable than compiled code."
It's mentioned there because that would be the usual expectation: an interpreter has different features, it usually enables stepping and tracing for example.
> observable to the running program
The interpreter is a part of the running program. In many Lisp systems the interpreter has slightly different semantics which are easy to observe: for example the interpreter usually will macroexpand code before each macro call. Where the compiler will do it BEFORE the code runs and it will do it once. The interpreter usually also offers hooks, which allow code to be executed, which changes the environment of each evaluation. Which may also easy to observe from the program. For example the interpreter could collect call statistics while running the code and the code could itself access that.
> it's not a property of the language itself
Lisp is as much a runtime as it is language. It's a combination of that. This makes the interpreter and compiler part of the execution environment and allows the program to do introspection, runtime code generation, observing its own execution in the interpreter and so on.
Common Lisp has a standard language spec, but it spans a room of very different implementations. The standard is very vague about the execution, it defines a bunch of interfaces (EVAL, ...) but says very little about their implementations. Garbage collection, a basic feature almost of every implementation isn't even mentioned.
The concrete language I'm using is not ANSI CL, but it is Clozure Common Lisp, CLISP, LispWorks, etc. - each with different implementation strategies. One might not use an interpreter, the next one has the interpreter more prominently. If you type to the SBCL REPL, everything by default gets compiled. If I type to a LispWorks REPL, everything gets by default interpreted.
> Some Lisp interpreters directly examine s-expressions
If they don't use s-expression, they are not a Lisp interpreter. Running off internalized s-expressions is the definition of a Lisp interpreter.
> I take it the answer is "yes" to my question "Are you drawing the distinction that it is only when an s-expression is paired with an interpretation that you get an AST?"
A Lisp interpreter does not need to use an explicit AST as an actual tree. It may just traverse the code according to the hard coded rules for the various built-in language constructs (function calls, sequence of calls, non-local transfers, quotation, ...) and build some data structures it needs.
> n Lisp, there are two levels of syntax: syntax classes from the source code, like symbols, numbers, lists; and syntax classes from the s-expressions, like what you go on to say.
In Lisp there are two levels of syntax: s-expressions as a data syntax. Numbers, symbols, lists, arrays.
Then there is the syntax of the programming language Lisp and its language constructs.
For example the EBNF-like syntax for the special operator LET is this:
let ({var | (var [init-form])}*) declaration* form*
> My experience observing lispers communicate is that they seem to assume nobody else is familiar with the details of Lisp and how it works!
I had the impression that you don't know what a Lisp interpreter actually does, how it is implemented and that it is a part of a Lisp system - and thus of the running program.
Take for example the LispWorks interpreter/compiler.
CL-USER 1 > (defvar *hooklevel* 0)
*HOOKLEVEL*
CL-USER 2 > (defun hook (x)
(let ((*evalhook* 'eval-hook-function))
(eval x)))
HOOK
CL-USER 3 > (compile 'hook)
HOOK
NIL
NIL
CL-USER 4 > (defun eval-hook-function (form &rest env)
(let ((*hooklevel* (+ *hooklevel* 1)))
(format *trace-output* "~%~V@TForm: ~S"
(* *hooklevel* 2) form)
(let ((values (multiple-value-list
(evalhook form
#'eval-hook-function
nil
env))))
(format *trace-output* "~%~V@TValue:~{ ~S~}"
(* *hooklevel* 2) values)
(values-list values))))
EVAL-HOOK-FUNCTION
CL-USER 5 > (compile 'eval-hook-function)
EVAL-HOOK-FUNCTION
NIL
NIL
CL-USER 6 > (hook '(cons (floor *print-base* 2) 'b))
Form: (CONS (FLOOR *PRINT-BASE* 2) (QUOTE B))
Form: (FLOOR *PRINT-BASE* 2)
Form: *PRINT-BASE*
Value: 10
Form: 2
Value: 2
Value: 5 0
Form: (QUOTE B)
Value: B
Value: (5 . B)
(5 . B)
Thus I can write a user level program, which uses the Lisp interpreter to observe the program running. It tells me what forms it is running and the result values.
The program itself could now for example observe its evaluation level:
CL-USER 9 > (hook '(list *hooklevel* (list *hooklevel*)))
Form: (LIST *HOOKLEVEL* (LIST *HOOKLEVEL*))
Form: *HOOKLEVEL*
Value: 2
Form: (LIST *HOOKLEVEL*)
Form: *HOOKLEVEL*
Value: 3
Value: (3)
Value: (2 (3))
(2 (3))