Hacker News Reader

Comment by duped | Hacker News Reader

The tokenization speed issue is already addressed by the language server protocol, which delegates syntax highlighting to language servers via the "semantic tokens" API. Language servers can choose to implement this however they want, and the API allows for incremental additions.

I think this is a much better approach that baking tree sitter into VS Code and continuing to use TextMate grammars (or tree sitter specific grammars) to add syntax highlighting. That way it can be applied to any editor that supports LSP integration. Really the world would be a better place for IDEs and text editors if we could just get LSP more standardized, and VS Code's dominance is a decent place to start with it. I'm tired of relying on editor and IDE authors for language support.

View on HN

> The tokenization speed issue is already addressed by the language server protocol [...]

It's not really addressed. The semantic tokens API is intended for semantic highlighting:

> Semantic tokenization allows language servers to provide additional token information based on the language server's knowledge on how to resolve symbols in the context of a project.

Abusing the semantic tokens API for syntactic highlighting is slow, unnecessarily complex (why do I need to implement a language server just to do syntactic highlighting?), and only a partial solution (still need a TM grammar, still don't get correct code folding, etc.).

I believe I've read in various places that the difficulty of doing syntax highlighting fast enough is precisely why it isn't done via LSP.

Am I right in thinking that the Semantic Tokens part of the spec (currently) falls short of saying "this is for all your syntax-highlighting needs, please go ahead and implement full-blown syntax highlighters using it"

I definitely agree with you that the more we can share between different editors/IDEs the better.

> I think this is a much better approach that baking tree sitter into VS Code

they're implementing both, with tree sitter being 'dumb' version of LSP syntax highlighting: https://github.com/microsoft/vscode-anycode

Oh thanks for that link, that's really interesting.

Can someone explain the paragraph below -- I thought "this is an invocation of a function named bar" was what we mean by semantic information:

> All features are based on parse trees and there is no semantic information - that means there is no guarantee for correctness. Parse trees allow to identify declarations and usages, like "these lines define a function named foo" or "this is an invocation of a function named bar"

It uses heuristics to derive semantic information from the parse tree without doing a full semantic analysis.

Is "semantic analysis" in the sense you used it similar to "type checking"?

Semantic analysis is a broader term that (for languages with a type checker) includes type checking. https://cs.lmu.edu/~ray/notes/semanticanalysis/

What's "dumb" about Anycode is its semantic features such as "go to definition", code completion, etc., not its syntax highlighting. Alas, Anycode is a separate project.

I was curious about the implementation, and was surprised to find:

- they’re importing query files, with a .scm extension, in TypeScript

- they’re using esbuild to handle those imports

It’s surprising to me that the Microsoft, who created and maintains TypeScript, is using an alternative TypeScript compiler… in part to query semantic information from TypeScript.

Reply on news.ycombinator.com