Project Lamdu: A Live Programming Environment and Language
lamdu.org
lamdu.org
So what I wonder is this: is there a migration path instead? If the goal is to ultimately work on ASTs instead of treating software as a giant string, is there a way we can make incremental transformations to languages to move in that direction.
So far what I've considered is this:
1 - Automated formatting such as gofmt which eliminates any decisions around whitespace to culture people towards treating code as data rather than prose
2 - Language-aware editors that constrain user input to valid states based on 1
3 - Plugins for version control systems to optionally do diffs on ASTs rather than plain files
4 - Compilers that can operate on ASTs instead of strings
5 - Checking in ASTs rather than strings based on 3 and 4
6 - Updating editors 2 to operate on ASTs since 5 makes it reasonable to always work on ASTs rather than text
7 - Creating new styles of code editors that visualize and manipulate ASTs in ways that are independent of their text representations
There's probably steps I'm missing. I'm only just getting in to this space so I'm ignorant of most research that has been done in this area.
When you get right down to it, "working on ASTs" basically means working in lisp. The idea is you have a canonical, external format for the AST; it might as well be s-expressions. The trouble is that lisp culture is highly resistant to any other syntax than raw s-expressions. They say things like "making alternate syntaxes is a rite of passage, but everyone gets used to the parentheses in the end."[1]
Racket alone encourages the creation of alternate syntaxes, and provides the tools to make it easy. You can have any syntax you want, and yet never be hamstrung by syntax decisions because you can always make a new version of your #lang, and it'll all be nicely compatible because it gets turned into s-expressions anyway. This isn't just theoretical - you can for example already write Python in Racket [2], except with full access to all of Racket's libraries as well as Python's. This turns the social aspect to its advantage. Imagine if Python had been written in Racket to begin with - the 2->3 nightmare would have been a non-issue.
I hope that eventually, all languages will have swappable syntaxes with a canonical s-expression form, and be distinguished purely by their fundamentally incompatible semantics. We may find we need far fewer incompatible silos that way.
Incidentally, and I don't think it's a coincidence, Racket's IDE is the furthest along the non-textual route of any "real" language I've seen. It's very next-gen - you can paste an image file directly into source code, for instance (try it, (define my-image <pasted image>) actually works!).
[1] I view all "rites of passage" with suspicion. Like the "rite of passage" of wielding 'rm' incautiously, they usually indicate a human factors failure.
Having Python syntax doesn't give you access to all the Python libs. Lot's of things need the Python runtime and also its support for non-Python extensions.
Literally hundreds of languages had been implemented on top of Lisp over the years. My Symbolics Lisp Machine had already in the 80s full C, ISO Pascal, Fortran, Ada and Prolog implementations written in Lisp. I was running on my Lisp Machine the original X11 server written in C and TeX written in Pascal...
Python is also not an alternative syntax on top of Racket or Lisp ... it's a different language.
It's just one monolithic component trying to take on type systems, effect systems, syntax, version control, compiling, editing, distributed execution, etc.
All of these problems taken decades just to get to the shitty everything-is-text/worse-is-better state we're in now. I'm skeptical that any single team can solve them all at once, even with the most radical vision and competence.
This is in contrast to the current state of development, which was driven by Moore's law and the rapid improvement of single-core processors to consist mostly of imperative programming. As soon as we started hitting a wall in single-threaded CPU improvements and moving to multi-thread and multi-cores, the imperative programming model began to rapidly show its limitations. After all, Von Neumann meant for his architecture to be a prototype, and we went ahead and built an entire industry on top of it.
Leaving everything in the development ecosystem up to market forces has left us in a state that is entirely too complex. There is far too much accidental complexity in developing even the simplest applications today.
If we purposefully design smarter systems, maybe we can cut out layers out of the stack entirely, so worrying about things like distributed execution is a thing of the past for 99% of programmers.
Actually I disagree. If your layers represent conceptual abstractions, and not arbitrary divisions of your code stack leaking details from below, then _more_ layers is actually a good thing.