Tree-Sitter | Hacker News Reader

Tree-Sitter

tree-sitter.github.io

112 pointstosh17 comments

I use neovim, the tree-sitter support is great especially with nvim-treesitter which means I can just run `TSInstall <parser name>` and I'm done.

Writing a tree-sitter parser is not that easy though. I wrote tree-sitter-templ to parse templ files and I still have a hard time identifying some parsing errors, especially since I had to implement a custom external scanner which in my experience was the source of a lot of bugs.

[0]: https://github.com/vrischmann/tree-sitter-templ

It's not easy, but after a while it does become somewhat meditative. I think the TypeScript nature of it helps a lot.

I picked up Neovim at my last job and got a decent start on a Tree-sitter parser off hours for the WinSCP batch programming language of all things. But then I couldn't get it to actually install correctly into Neovim, and I switched jobs before I figured it out. :( EDIT: actually, hey, your instructions on installing are really clear! I wonder if I can use them here to help guide me again.

It was really cool, though! Love parsers. https://github.com/hiAndrewQuinn/tree-sitter-winscp

Neovim supports treesitter based selections and "jumps", which I think are awesome

https://www.reddit.com/r/neovim/comments/r10llx/the_most_ama...

https://github.com/RRethy/nvim-treesitter-textsubjects

Helix also supports them by default

I've tried to write a parser using tree-sitter multiples times, but it's complicated enough that I've found it to be simpler to write my own parsers. Of course as a mere amateur, my parsers don't have certain nice to have features like good error messages, or good incomplete syntax recovery. I wish tree-sitter was easier to use.

I didn't have much trouble writing tree-sitter parsers after taking a look to examples but I also resigned to writing my own parsers instead of using it.

There were two concerns I couldn't shake from using it.

1. In C-like languages you usually can do forward declarations, so you can opaquely define something, use it and have it properly defined later, usually due to how things need to use other things in their declaration. You can't cleanly do this in tree-sitter, you will need to handwrite some C code for this on top.

2. Even though the resulting parser is a single C-file, it still depends on some tree-sitter library and building it requires nodejs and npm packages. This means that anyone updating my C/C++ codebase has to also understand JS tooling. I would rather not have such dependency. If tree-sitter could have standalone tools that could be built only using either a C or C++ compiler, I would not have such a problem.

I tried to write a vhdl treesitter grammar and I too found it hard. I got something working enough for decent highlights but not a complete grammar by any means.

Related. Others?

How to Get Started with Tree-Sitter - https://news.ycombinator.com/item?id=36106421 - May 2023 (67 comments)

Topiary: A code formatting engine leveraging Tree-sitter - https://news.ycombinator.com/item?id=36008139 - May 2023 (33 comments)

Rust-sitter: Define your entire tree-sitter grammar in Rust code - https://news.ycombinator.com/item?id=34892916 - Feb 2023 (14 comments)

Combobulate: Structured Movement and Editing with Tree-Sitter - https://news.ycombinator.com/item?id=34601089 - Jan 2023 (15 comments)

Writing a TLA⁺ tree-sitter grammar: my foray into free software - https://news.ycombinator.com/item?id=34356429 - Jan 2023 (3 comments)

Emacs: Feature/tree-sitter merged into master - https://news.ycombinator.com/item?id=33715321 - Nov 2022 (89 comments)

Tree-sitter grammar for org-mode - https://news.ycombinator.com/item?id=30945926 - April 2022 (31 comments)

How to write a linter using tree-sitter in an hour - https://news.ycombinator.com/item?id=30822544 - March 2022 (11 comments)

How to write a tree-sitter grammar in an afternoon - https://news.ycombinator.com/item?id=30661127 - March 2022 (9 comments)

Tree Sitter and the Complications of Parsing Languages - https://news.ycombinator.com/item?id=29327424 - Nov 2021 (134 comments)

Diffsitter: A tree-sitter based AST difftool to get meaningful semantic diffs - https://news.ycombinator.com/item?id=27875333 - July 2021 (53 comments)

Tree-sitter: an incremental parsing system for programming tools - https://news.ycombinator.com/item?id=26225298 - Feb 2021 (134 comments)

Tree-sitter: new incremental parsing system for programming tools (2018) [video] - https://news.ycombinator.com/item?id=21675113 - Dec 2019 (28 comments)

Tree-sitter – a new parsing system for programming tools [video] - https://news.ycombinator.com/item?id=18213022 - Oct 2018 (25 comments)

How does TS compare to using semantic highlighting in LSP?

A large number of LSP servers and related software use tree-sitter as a backend.

That's not possible, as Treesitter doesn't do for example type-checking (unification). It can be used as the parser to generate the AST with which you're actually doing the work.

Agree, though not sure that's what the parent meant. I think they meant:

1. Some LSP implementations use Tree Sitter as the parser in their implementation

2. But it's only part of the implementation. It just generates the parse tree, on which various other LSP features are built, e.g. enhanced syntax highlighting, go to definition, ...

Tree Sitter is pretty good for (1) because it was built specifically to cope with code that is (a) changing rapidly and (b) is, by default, not valid. So it has good error reporting and recovery.

Having tried both with Neovim, I ended up going with just LSPs instead of using tree-sitter.

With Treesitter you have another parser which is redundant at best and inconsistent with the LSP at worst.

Not sure I understand your point.

LSP is a protocol and tree-sitter is a parser generator. They're kind of orthogonal concepts; a tree-sitter parser couldn't ever be used directly in place of an LSP server, but an LSP server may well make use of tree-sitter as a first step for extracting information from the code and keeping it in sync. If it doesn't it'll have to come up with some other way of parsing the code in any case, so I don't see how it could be said to be redundant or inconsistent.

Of course, tree-sitter's thing is how universal it is. There's parsers for tons of languages, and you can work with them all using the same API, though you're on your own for attributing semantic meaning. Most popular languages have language-specific tools (e.g. `libcst`) which are usually more powerful for that specific language, so they'd probably be better starting points for building a language-specific LSP server which I imagine is the common case.

> Not sure I understand your point.

The problem is using Treesitter (for syntax highlighting and "semantic movements") and an LSP at the same time. So if your language has a LSP, using Treesitter additionally is redundant at best and introduces inconcistency at worst.

I'm not talking about using Treesitter as the parser for the LSP.

> Most popular languages have language-specific tools

I'd say even less popular langauges like Coq^H^H^HRocq, Lean 4, Koka, Idris, Unison, ... have their "own" tools, I do not know of a language that uses a Treesitter parser in its LSP, but I do know about tools like https://semgrep.dev/ (written in OCaml) and Github's code search which use Treesitter.

You're forgetting that treesitter is much, much faster than LSPs.

It's slightly worse but a lot faster and better at dealing with spelling/syntax errors. Also, the LSP can render inline error messages and line markers on top of TS highlighting so not much information is lost by just using TS highlighting everywhere.

Reply on news.ycombinator.com