Hacker News Reader

Comment by thih9 | Hacker News Reader

What would you consider a perfectly solved problem in this case? I.e. how is current development experience bad and how it could be better?

View on HN

In a perfect world of emacs/tree-sitter, I would imagine tree-sitter to be a single minor mode. You don't need 'java-ts-mode' and 'c-ts-mode'. Just a single 'treesitter-mode' toggle and it will call the treesitter binary to do the hard lifting.

I guess the reality just isn't so rosy.

A `treesitter-mode` is a so-so idea, because the way different languages are interacted with is different. The way lisp sexp handle in an IDE is different from Python's syntax. It doesn't make sense to try and have one mode that handles both - it is better to have many modes that interact with the common data structures that treesitter provides and then starts to provide specific convenience functions.

Even if all it was supposed to do is syntax highlighting, different languages are usually highlighted differently and presumably the person maintianing the bindings would need to be a language expert with strong opinions. The options are they either maintain their own mode or a module in `treesitter-mode` which is basically 2 ways of describing the same situation.

Although to address your other point it is quite funny that after 40 years and 29 versions of experimenting in OS design, GNU Emacs is starting to implement syntax highlighting properly for its system text editor.

A treesitter-mode would make sense for organizational reasons and user-experience. But under the hood it would be just a proxy-mode which figures out the language and load the appropriate treesitter-sub-modes. And maybe in a more advanced version it could also mix modes for different languages.

But you're describing Emacs. Emacs figures out the language and loads the appropriate sub-modes. It even calls them modes. Modes are mixable.

We're in a transition period while everything is rewritten to use tree-sitter. In a few years all the default major modes in Emacs are probably going to be tree-sitter based - unless the maintainers believe them to be better than tree-sitter.

> Emacs figures out the language and loads the appropriate sub-modes.

Not really. You setup hooks, usually based by fileextension, and emacs execute the hooks. Emacs itself has no understanding of languages. And this will not work well when there is no hook executed.

> Modes are mixable.

You can load multiple minor-modes, but there is always just one major-mode per buffer, and languages are usually major-modes. But ok, treesitter can be handled as a minor. But it still needs explicit support. You cannot give treesitter control over a range of text, let if figure out the language automatically and let it make it's thing. This needs explicit support.

> GNU Emacs is starting to implement syntax highlighting properly for its system text editor What do you mean?

It is a classic joke. Emacs is often a little behind competing text editors (Vim has better keybindings IMO, things like Visual studio have better IDE support, Atom text editor turns out to have better syntax highlighting, etc, etc).

So, the line of the joke goes, it must not really be a text editor because it is not very good at it and has a wild array of other capabilities. Like "M-x dunnet". Must be an OS.

Of course the reality is that Emacs just copies good ideas from other people. A pretty normal Emacs setup uses Vim keybindings, LSP and now tree-sitter to get a good experience programming.

> Of course the reality is that Emacs just copies good ideas from other people. A pretty normal Emacs setup uses Vim keybindings, LSP and now tree-sitter to get a good experience programming.

Too bad other people don't copy good ideas from Emacs. Why other text editors/processors have no equivalent of view-lossage is a mystery to me. Also, where-is, describe-key, insert-char, transpose-chars and lots of other very useful stuff...

I'm not the GP, but personally I'd consider syntax highlighting perfectly solved if it was handled in the language server. That way we could have highlighting performed by 100% accurate parsers instead of approximations. It wouldn't be as fast as in-editor simple tree-sitter highlighting, but I don't see any issue with my text appearing uncolored at first and gaining color as my editor polls the language server as I type.

Currently language servers support semantic highlighting which is close, but as far as I can tell, it seems meant to be supplemental to in-editor local highlighting rather than a full replacement.

You would only get the top half of your code highlighted because a compiler doesnt usually continue when encountering syntax errors due to partial code

> You would only get the top half of your code highlighted because a compiler

Your comment implies a language server taps into the language's compiler/interpreter. That is a popular misconception. Almost all LSP severs don't actually use the compiler backend of the language they are servicing. All the LSP server implementations I've seen at least just use a static parsing approach (or even worse i.e. just a tokenizer) similar to what Treesitter does. It is just not limited to a single file.

That's why I still think Treesitter has the potential to not only improve syntax highlighting but also to simplify language servers or even -- in the long run with some extensions of the the language modes -- replace it altogether.

But some do. Most notably clangd. Still clangd tries parse broken code using recovery heuristics (which can still be useful in a compiler to produce decent error messages).

> Your comment implies a language server taps into the language's compiler/interpreter. That is a popular misconception.

But it is a misconception which framed the discussion, not a misconception of the answerer. The claim was that LSP was

> performed by 100% accurate parsers instead of approximations

If the code was valid at some point (e.g. when opening it) it could remember the old colorings of the rest of the code.

Though handling opening and closing a string and still ending up in an illegal state would need some more logic.. Maybe not worth it.

> If the code was valid at some point (e.g. when opening it) it could remember the old colorings of the rest of the code.

If you continue down this line of thinking a few more steps, eventually you produce tree-sitter.

That's sort-of already the case whenever we start a string in any language that supports multi-line strings. You type the first " and then the rest of your file gets re-interpreted as a string. It is annoying but I still think its a worthy trade-off, because I spend a lot more time reading code than being mid-edit with invalid syntax.

Syntax errors are not limited to just typing a quote

I never said they were. I'm just pointing out that we already experience the same problem in the case of quotes without the world ending, and I don't think expanding the problem to all syntax errors will make it much worse.

It would make it significantly much worse. Because every incomplete statement that you're just writing would break all syntax highlighting right after your cursor. And if you use any modern editor or IDE this doesn't happen.

> You would only get the top half of your code highlighted because a compiler doesnt usually continue when encountering syntax errors due to partial code

Sounds like a feature - don't need to scan for tiny squiggly red lines, or examine some tiny font output in a status bar somewhere.

I'd love it.

But you want syntax highlighting performed by approximations, because when editing your code is only approximately valid.

It would be even better if the code was always valid because we only edit the AST itself, like what paredit does for Lisp modes.

No.

(The code being valid s-sexps doesn’t even guarantee it’s syntactically valid after macro expansion, but even if it does this is a bad idea.)

One interesting instance of semantic highlighting occurs in the Lean 4 language. It has extensible syntax whose parsers can be expressed in the language itself. Since parsing and evaluation are intertwined, highlighting can only really be effectively achieved in an LSP lest you re-implement the entirety of Lean.

Reply on news.ycombinator.com