Sapling: Experimental vi-inspired editor where you edit code, not text | Hacker News Reader

Sapling: Experimental vi-inspired editor where you edit code, not text

github.com

191 pointstodsacerdoti47 comments

Since this project ended, we've have had quite a lot nice development in this front. Biggest highlight maybe is Combobulate for emacs, providing treesitter powered editing. Neovim also has treesitter and some plugins[2] to work with it. My editor of choice, Helix, also has treesitter based commands built-in[3]. All of these afaik have been released after sapling

While it now seems like this mode of using TS/structured editing to extend traditional editors will be the dominant way, I think it still might be useful to keep thinking what an editor that embraces structured editing could look like.

[1] https://github.com/mickeynp/combobulate

[2] https://github.com/nvim-treesitter/nvim-treesitter/wiki/Extr...

[3] https://docs.helix-editor.com/usage.html#navigating-using-tr...

I've been using and enjoying Helix for a while now, but I don't use any structured editing commands. I should definitely try some out!

I had reasonable experience with structure editors (worked on one long time ago, for beginner programmers) and at least in their naïve form they are a bloody pain to use, and far slower to get information in than a halfway decent touch typist. It's possible some improvements might make things better, but it's really hard to see. Happy to be proven wrong!

Don't forget the language intelligence core being built from the ground up as a universal engine for semantic editors!

[4] https://github.com/bablr-lang/

This is one of the hills to climb when learning a lisp. Seasoned lispers don't edit text, or traverse text; they edit and traverse lisp forms (AST nodes). In my experience, vim/neovim work very well for this, due to their notion of "text objects". With a couple of plugins, text objects end up aligned directly with the lisp AST, but even better is that the AST is visual and tangible, since it's encoded in the text.

It makes sense to benefit from this sort of thinking in other languages, but this is just another reminder, to me, of the beauty of lisps.

A fun language idea would be to take the exact semantics of an existing language, like C++, and encode them within a lisp, losslessly. I suspect it would end up quite verbose. However, add in a macro system and then you can start building a DSL around the original language, keeping all the same underlying semantics.

Isn't that a bit what hy (https://hylang.org/) tries to do ? AIUI it is a lisp interacting directly with the AST of Python, allowing seamless interop: Python modules can be used from hy and vice versa, everything is transparent.

I'm not sure if it's doing exactly that, but it looks like it could be! Very neat. It'd be nice to see it for other langs, if Hy proves viability in the approach.

Other commenter mentioned Hy, but what about Clojure? It is a lisp on top of Java, and all of its variants (Clojurescript -> Javascript, etc). Then there's Fennel (Lua) and others I can't remember right now.

Clojure sits on top of the JVM and allows for Java interop, but the language itself does not losslessly encode all of the semantics of Java. Same goes for ClojureScript and the other dialects.

The difference here is that the lisp's AST would need to map 1:1 with its underlying language.

I find this idea interesting, because it's my impression visual programming is generally considered less powerful than textual programming, but it sounds like the described setup where you visually edit an AST could be a sort of middle ground.

I agree completely; this is why I think a lisp would be a much better scripting language for a text editor than vimscript or lua. Although I use vim at an advanced level, I have never written a plugin for this reason. Emacs offers a much better scripting environment with elisp.

I know you can write Vim configs in Fennel (a LISP that compiles to lua), would that work for plugins?

Good question; although I am being told that Fennel still requires to keep lua shenanigans in mind.

For Emacs users, there's Taylor Campbell's `paredit`:

https://www.youtube.com/watch?v=D6h5dFyyUX0&t=51s

https://paredit.org/

https://www.emacswiki.org/emacs/ParEdit

Emacs kids these days switched to using smartparens and now Parinfer

Lispy https://github.com/abo-abo/lispy

When you're too cool for temporal modal editing so you invent spacial modal editing.

Been using that for a while and yes it is the best parts of vim with the best parts of Emacs.

The main selling point is that I can now edit lisp code on my phone with the thumb key keyboard... Come to think of it I can probably use a physical corded keyboard now too...

No, turns out physical chorded keyboards are a lot worse than I remember for code.

Circa 2000, some people were using the Twiddler one-handed chorded keyboard with wearable computing. I'm not certain this was for coding, but Bradley Rhodes was using his wearable heavily, with Emacs as the primary UI in his head-up display.

They are OK for text input, it's the extra symbols you need for coding that are the issue. Even the () for lisp are second class characters.

I also recommend symex[1]. Although it is more “locked-in” to s-expressions than other solutions (which takes some getting used to at first), I find that for me this is exactly what makes movement feel much more intuitive and editing much more precise.

The one thing I don’t like is that symex depends on so many other plugins (especially Evil, which I am trying to swap out with the more lightweight meow), but this will apparently change soon. They are also working towards support for non-Lisp languages via tree-sitter, but I don’t know how well it works.

[1]: https://github.com/drym-org/symex.el

I think the core thought in "But Why" kind of gives away the problem this will face. The idea is that you will lose time as you "move between valid programs." A problem, though, is most "valid programs" are still not valid solutions to what you are working on.

Consider, most of us work in rather phonetic languages as we type day to day. Surely it would be easier if we worked instead in languages where you did not have to do this. Let me type only valid sentences as I move from thought to thought. Surely that would be better? Why, then, do we not do that?

Of course, I don't know why. I suspect there is a good deal to inertia style thoughts that explains this. But it is a good hint that the idea that you can break apart from symbols that need more symbols for context to build meaning is not one that really anything has fully converged to.

I always think of this when I edit python and my code is indented wrong... not a fan by structuring by indentation, but it doesn't seem to be a problem for most people.

Cards on the table, I'm a lisp fan, so I'm probably the closest you can get to directly editing an AST for my dream program. I /like/ the idea of it. I just don't think it is a holy grail endpoint of program editing.

Oddly, I do think there is something to making sure all of your data has a literal representation in the code, if you can. Such that I do think being able to directly interact with data is huge. I also love when you can symbolically work with programs.

But, yeah, the overwhelming evidence is that this just isn't necessary for things. Both in communication, and in program specification.

I think you're thinking about it subtly wrong. A properly built semantic editor won't force you to move between valid programs in that sense. It will be comfortable decomposing programs into fragments.

For example if I decompose the sentence "I press play" into words devoid of context, "I" still pretty much means what it did but "press" could now as easily mean pressing shirts or pressing an advantage, and "play" could be what children do, not the implied "play button".

A good semantic editor for that sentence would allow me to detach the word "play" without losing the context that the meaning is "play button". Similarly a good semantic code editor would let me do something like pull two non-adjacent class methods out of a class and into a clipboard, then drop them into another class. If you did this this with text, the editor would not understand the clipboard content as class methods are (usually) not parseable outside a class.

Symbols are not context free, though. And words being compound symbols means they are heavily dependent on context. Pretty much period.

You can argue that I should be able to copy a method invocation and move it, but I confess I don't even really understand what that would mean or how it would help me. More, you are fixed in a global evaluation sense of the work, now. What if I want to take a formula in code, evaluate its derivative, and then use that for evaluating input? (That is, the exact same code depends on where it is

But, again, my point was not that we can find the reason it doesn't work. I don't know the reason it doesn't work. I don't even know for sure that it can't work. What I do know is that no other communication has distilled to this level of editing. Happy to be proven wrong.

I had meant to edit my post to refer to this all as "a" problem that this idea will face. Specifically that no other editing of anything that I am aware of happens at this level. There are some neat composition tools that can put together some scenes in this way, I think. Such that maybe there is more of a generalization there than I think there is. Seems very unlikely, though.

Now, oddly, I do think software is best made to enable this style of construction for what it controls. Easy example would be a level editor in a video game. That makes surprisingly little sense to do in a symbol based editor, as opposed to a palette based visual editor. Such that I think I'm open to the idea, at some level. Symbol manipulation is just not that level, for most things.

Sapling is the first AST editor that works how Ive imagined it could. I'd love to leave all the whitespace wrangling behind and move only between AST nodes in an efficient way. This was a fun editor to try and I think there is something to learn here still to make treesitter editors more powerful/efficient.

"where you edit code not text"

gif shows the user editing text

I don't know why I haven't thought of this before, but I kinda love the idea of an AST Editor.

A buddy of mine and I wanted to experiment with such a thing back in ~'85. I started by looking at the "elvis" editor code, trying to wrap my head around it, but I didn't really have a clear picture in my head about what a "modal AST editor" would do, just some vague ideas of an editor that operated on parsed syntax (AST wasn't on the horizon for us at that point). I was in HS at the time, and my buddy was working on a masters thesis, I think he was too busy and it was way too big a project for me to handle at that point.

One interesting thing about GPT&cetera is that in order to "cheat" by using it, you need to clearly describe the problem you are trying to solve. If we can get more people to design before coding begins, I think we're going to have a win in any case.

It’s a very interesting space. There are several experimental editors out there - they’re all rough as you might expect, but the possibilities are fun to think about.

I’m building a semi-structured editor for UI - I’ve described it as “vim for designers.” Just as an example of how varied the general idea can be.

I’m VERY interested in this vim for designers idea. Have any resources available to learn more?

Glad to hear there’s interest! Right now it’s still a small side project, but I could probably toss up a prototype to play around with.

I was working on a programming language for designers, and have recently been thinking that it just works better if you build against an AST directly instead of text.

The language is called Matry:

https://matry.design/

I'm also very interested. I have similar thoughts about the future of "design" (for a very wide-ranging meaning of the word)

Yeah the current tooling landscape is seriously underpowered.

I'm worse, I have thought about this many times but never did anything with the idea. My respect for the authors for actually trying to materialize it.

Here are some problems faced by semantic editors:

1. Bias. A lot of people think that if good ones haven't been built already, there must be some reason they can't be built

2. You really want the parser to report not just what types of nodes it found at particular locations, but what types of nodes it could have found at that location, because those are the types that will be valid for insertion there. A semantic editor needs to know because it should only be able to create programs that the parser would be able to parse.

3. Extensible parsers: To make a huge number of parsers that exactly match languages, you need to be able to quickly define parsers that are closely related to each other, and what's more the cost of using the derived parser must be closely related to the cost of using the original. Tree-sitter can derive languages at compile-time, but not at runtime leading to three major problems:

- 3.1 Runaway storage costs: the Tree-sitter grammar for JS is ~4mb, and the grammar for TS is ~12 MB and the grammar for TSX is also ~12mb and these grammars have things they shouldn't have in them like Ember's glimmer syntax, because that syntax is too poorly supported to justify paying the costs of copying the JS, TS, and TSX grammars (causing 28MB of new code duplication, in git no less).

- 3.2 Failed semantics: Because the JS runtime grammar is just code copied into the TS runtime grammar, tree-sitter can't distinguish which parts of TS programs are really just plain JS.

- 3.3 Umbrella languages: Because the grammar defines languages that are bigger than what you can actually use, the JS tree-sitter grammar will happily parse JSX even when it's invalid, which is to say a semantic editor would suggest inserting JSX even when it would break your program, which is clearly bad.

Python's predecessor, ABC used a structured editor.

One of those things that I always thought was a great idea, and it's fantastic to see someone actually working on it. AST editing really is the next step for modal editors IMO.

Such works might be quite useful for natural language AI infused code editing as there would be a coherent interface for the AI to give commands to and a smaller room of error (given that the action space is smaller than when a standard text editor is used)

This was something that I was thinking of for a while now, and now seeing what it looks like, I don't quite like it and I'm glad I don't have to try and implement it to see how it works! :D

Great work though, this is a super cool project.

I also had a similar thought, decided to implement it as a neovim plugin[1] to see what it feels like. Turned out okay! I use it daily now, but I think the important part is it's just a "mode" in an otherwise normal editor so I'm not forced to only navigate by AST nodes.

[1] https://github.com/gsuuon/tshjkl.nvim

Reminds me of Steven Reiss's "PECAN" system he developed at Brown University in 1984, which he gave a talk about and gave us a copy of the Sun version to play around with at the UMD Human Computer Interaction Lab around 1985 or so.

It seemed like an interesting idea, possibly useful for teaching programming to students who had never coded before (especially the data structure visualization and algorithm animation system BALSA), but essentially pretty frustrating and klunkey in practice.

Of course we have a hell of a lot more computing power to piss away at the problem now, but it's still fundamentally at odds with how professional programmers code and how programming languages are designed (except for Lisp of course), and isn't a widely used approach for many practical reasons.

https://cs.brown.edu/~spr/papers/pecanpaper.pdf

>Graphical Program Development with PECAN Program Development Systems (1984)

>Steven P. Reiss, Department of Computer Science, Brown University, Providence, RI 02912.

>Abstract:

>This paper describes the user's view of the PECAN family of program development systems. PECAN is a program development system generator for algebraic programming languages. The program development systems it produces support multiple views of the user's program, its semantics, and its execution. The program views include a syntax-directed editor, a declaration editor, and a structured flow graph editor. The semantic views include expression trees, data type diagrams, flow graphs, and the symbol table. Execution views show the program in action and the stack contents as the program executes. PECAN is designed to make effective use of powerful personal machines with high-resolution graphics displays and is currently implemented on APOLLO workstations.

ChatGPT opines:

>Subsequent research into syntax-directed editors has revealed that they are less effective than initially anticipated. These editors have been thoroughly examined over the past decade, leading to insights about their limitations and practical challenges. Studies have shown that while syntax-directed editors can be useful in certain contexts, such as language definitions and programming environment development, they haven't gained widespread popularity due to various factors. These include issues related to usability, integration with other development tools, and accommodating diverse programming styles. Additionally, the complexity of implementing efficient and robust syntax-directed editors that smoothly integrate into existing development environments has also contributed to their limited adoption.

>Check out these search results to keep exploring.

https://link.springer.com/chapter/10.1007/3-540-17189-4_87

https://www.researchgate.net/publication/221330960_On_the_Us...

>On the Usefulness of Syntax Directed Editors. (1986)

>Bernard Lang, INRIA, B.P. 105, 78153, Le Chesnay, France.

>Abstract

>The intent of this position statement is to relate our experience with the actual use of a syntax directed editor (Mentor [5]) in different contexts: teaching, real software production, software maintenance and associated tasks, language design and development, development of programming tools and new programming structures.

>1. Introduction

>Originally the subject of this presentation was motivated by discussions to be found in the technical literature [1,2,3], and by the open questioning in less technical publications (as well as in private discussions) of the real usefulness of the syntax directed approach, its practicality and its high cost for allegedly low returns [4]. While the author does not dispute many of the points raised in the technical literature, he believes they emphasize too much the role of syntax directed tools for program creation, and thus lead to a biased assessment of their real usefulness. This is currently resulting in the marketing of sophisticated syntax directed editors, restricted to program creation, whose cost effectiveness may indeed be questionable.

>The ideas expressed here reflect only the point of view of this author, and are intended as a basis for discussion on the role of syntax directed editors. The personal judgements on various aspects of syntax directed environments are not meant as an assessment of the usefulness of research in those areas, but only as a tentative evaluation of the relative importance of these aspects for users, given the current state of the art and the author's own experience.

>This experience is based almost solely on the use of the Mentor system [5]. Though comparison of several systems would be desirable, the youth of this technology makes it rather difficult to acquire extensive practical experience with more than one of them.

[...]

I think Brett Victor's work on Learnable Programming, program visualization and live programming is a more modern and fruitful approach.

http://worrydream.com/LearnableProgramming/

>As you can see, live coding, on its own, is not especially useful. The programmer still must type at least a full line of code before seeing any effect. This means that she must already understand what line of code she needs to write. The programmer is still doing the creative work entirely in her head -- imagining the next addition to the program and then translating it into code.

[...]

>Here's an example. In many styles of programming today, when an application launches, it creates a large set of interconnected stateful objects. To see the effect of a code change, the application must be "relaunched" -- that is, its entire world is destroyed, and rebuilt again from scratch. How can we "create by reacting", continuously changing the code and seeing continuous effects in the flow and data, when there is no continuity between the application's state before and after the change?

>We can't. That's the wrong question. A better question is: How do we design a new programming model that does allow for continuous change? We already have clear hints.*

>Another example. Most programs today manipulate abstract data structures and opaque objects, not pictures. How can we visualize the state of these programs?

>Again, wrong question. A better attitude is to assert that we have to be able to understand the state of our programs. We can then ask: How do we design data structures that can be visualized? Can we invent data structures that are intended to be visualized? How do we move towards a culture where only visually-understandable data is considered sound? Where opaque data is regarded in the same way that "goto" is today?* Forward reference: Some work that I've done in automatic visualization of ad-hoc data structures will be published later this year, in collaboration with Viewpoints Research.

>In his influential essay No Silver Bullet, Fred Brooks makes the case that software is inherently "invisible and unvisualizable", and points out the universal failure of so-called "visual programming" environments. I don't fault Fred Brooks for his mistake -- the visual programming that he's thinking of indeed has little to offer. But that's because it visualizes the wrong thing.

>Traditional visual environments visualize the code. They visualize static structure. But that's not what we need to understand. We need to understand what the code is doing.

>Visualize data, not code. Dynamic behavior, not static structure.

>Maybe we don't need a silver bullet. We just need to take off our blindfolds to see where we're firing.

The problem here is, no one seems to have an answer for the question:

>What does an algorithm look like?

I've tried to use tools such as:

- https://www.blockscad3d.com/editor/ --- Blockly variant for OpenSCAD

- https://github.com/derkork/openscad-graph-editor

and after a while, any moderately complex task gets broken down into modules and you end up with the "wall of text" which ostensibly one is trying to avoid, or one has the visual version of spaghetti code:

- https://scriptsofanotherdimension.tumblr.com/

- https://blueprintsfromhell.tumblr.com/

That said, it's really nice when one edits a variable definition in a graphical tool and _every_ usage updates accordingly.

For visualizing data structures, although most data structures aren't designed to be visualized, there are tools to do this already.

https://github.com/hediet/vscode-debug-visualizer/blob/maste...

https://www.gnu.org/software/ddd/manual/html_mono/ddd.html#S...

I'm having trouble seeing what he has in mind that's different from the tools I listed.

somebody tell them about antlr4

Reply on news.ycombinator.com