While writing YAQL I accidentally discovered Lisp

So instead of having a separate meta-syntax to manipulate Lisp at the level of the AST, you simply write Lisp. Lisp is the language that's it's own AST. Combine this with a complete exposure of the meta-level of the language, and you get a language where many things that would be very expensive or impossible in other languages are relatively inexpensive and possible.

Smalltalk is nearly it's own AST. There are enough tools and exposure of the meta-level to enable things like Lisp macros, but it's still 2-3X harder than in Lisp. Also, Smalltalk Blocks by themselves are fairly powerful. As a result, people don't use Lisp style macros in Smalltalk even though they could if they really wanted to. (Implementing them would just take a few days.) The cost-benefit just doesn't work out.

(Yes, even Smalltalk is a kind of "Blub" compared to Lisp.)

in which the primary representation of programs is also a data structure in a primitive type of the language itself

This definition doesn't seem quite right to me. It seems like it would apply to C because C source code is a string which is "a data structure in a primitive type of the language itself".

Wouldn't a better definition be "a language in which the source code and the corresponding abstract parse tree have identical representations" ?

Slamming out a halfway decent parser is getting easier than it used to be with things like Parsec, and JSON support is nice enough in the languages I work in that I tend to just use that now, but in the 90s I went to this solution several times. (XML, on the other hand, wasn't much help. It did indeed take care of the real grunt work of processing but what was left over still took a lot of love to translate into the native idioms of the program you were in, what with the attributes and the mixed content and whatnot.)

Lisp feels powerful because you can manipulate programs as if they were data. This is easy in Lisp because your programs are already expressed as a tree, using a simple data model (cons cells).

In most other programming languages, the program first goes through this generally very hairy and complicated parser. The parser is not usually accessible from the language itself, which means you can't easily get your hands on an AST. Even when it is (like in Python: http://docs.python.org/library/ast.html) the AST is a complicated data structure that takes a lot of work to make any sense of.

But this doesn't mean that the solution is to write programs as raw ASTs! Programming language syntax is nice because it lets us write things in ways that are convenient for humans. Most humans would rather read 1 + 2 - 3 * 4 than (+ 1 (- 2 (* 3 4))).

I strongly believe that the way to bring the power of Lisp to other programming languages is to provide much easier access to parsers and ASTs, and better APIs for them. If we could accomplish that, we could keep writing syntax that works for humans, but have easy access to data structures so that the AST can be very easily inspected, manipulated, generated, compiled, interpreted, or any of those things that Lispers love to do.

When I say "better APIs," I mean that you should have lots of flexibility for how you want to consume, process, and store the AST. You should have SAX-like stream processing and data structures like DOM (but something that binds more naturally to programming languages, so you can say foo.bar instead of foo.getElementById("bar")). There should be a convenient serialization format for the AST, and there should be a nice way to specify the schema of the AST (like a more generic version of http://scottmcpeak.com/elkhound/sources/ast/index.html).

You might sense this coming, but I've been working for a long time on what I consider a major solution to this problem. There are two parts to it.

upb (https://github.com/haberman/upb/wiki/) is a standalone and very minimalist implementation of Google's Protocol Buffers serialization format. I believe Protocol Buffers are an ideal data model for ASTs. Forget for a moment that Protocol Buffers is a serialization format and consider that you can use their schema language to specify something like:

    message Expr {
      // Both operands can be either a raw value or another expression.
      double val1 = 1;
      Expr expr1 = 2;

      double val2 = 3;
      Expr expr2 = 4;

      // The operator that should be applied to the two values.
      enum Operator {
        PLUS = 1;
        MINUS = 2;
        MULTIPLY = 3;
        DIVIDE = 4;
      }
      Operator operator = 5;
    }

The second part is my somewhat dormant project Gazelle (http://www.gazelle-parser.org/). I mean to resume it once upb is ready. Gazelle uses most of the same parsing theory as ANTLR, but instead of generating code for every parser it has a more VM-like approach. And as its output? You guessed it -- upb and Protocol Buffers. You can consume the parse tree in a streaming way (SAX), as a data structure (DOM), you can serialize the AST itself as a Protocol Buffer.

tl;dr: Programming languages are just a human-friendly serialization of an AST. Parsing is simply tree traversal. Protocol Buffers are simply a universal way of specifying and manipulating trees. And I'm working on an attempt to unify it all into a tiny but powerful library that aims to bring the power of Lisp to all programming languages.

When trying to design my own languages I kept needing to handle special conditions and it seemed all roads lead to either Lisp or Smalltalk. It seems those 2 are fundamental paradigms.

I call shenanigans.