Hacker News Reader

Comment by avodonosov | Hacker News Reader

Anyone knows, why github code navigation, which is based on tree-sitter [1], supports only subset of languages for which tree-sitter has parsers [2]? For example, Common Lisp is not supported. I asked github [3], but they are silent.

1 - the "search based" navigation is based on tree sitter https://docs.github.com/en/repositories/working-with-files/u...

2 - the list of parsers in the official docs: https://tree-sitter.github.io/tree-sitter/

3 - https://github.com/orgs/community/discussions/55704

View on HN

While we're talking about this, I've been quite disappointed by the new GitHub file browser: the new features look neat, but the core file browsing experience is significantly worse for me because the new UI is much more complicated and every action seems to do way more work.

I couldn't immediately find a way to opt out; I wouldn't mind the new file browser so much if I could drop into it on demand instead of having it forced down my throat.

Click on your profile in the top-right, and then in the resulting menu, click on 'Feature preview'. You might be able to disable it from there.

It’s not there anymore, presumably because it graduated from beta earlier this month: <https://github.blog/changelog/2023-05-08-the-new-code-search...>

It was an optional opt-in labs feature that I tried for an hour then turned off, but now they force it.

The new UI is more busy for no good reason. Sidebar makes sense in a local IDE, but over web every time you expand a subtree it takes ages anyway. And because it merely exists it is distracting and makes pages load slower...

I’ve had few (some, but minimal) problems with the new file tree browser, but the actual individual file view has become a minefield. It hijacks find in page, presumably to enhance it with semantic relations, but it breaks frequently when navigating. Previous searches (cmd-G) will fall back to the browser’s native search. If you’re lucky, you searched the same term in both, and if you’re luckier still the search term is in the top N lines of code (because the code itself is lazy loaded I guess?). Most of the time I’m not lucky and I have to start over each time I view a different file. Even though it feels more like a “single page app” than it previously did, I lose all of that state every time I do anything.

I get what they’re trying to do, and in some ways it’s clearly beta growing pains, but in other ways it’s clear they’re fighting the browser environment in ways that don’t and probably shouldn’t work.

I ended up blocking keyboard events from a userscript to deal with the keyboard hijacking. This combined with opting out of all the "feature previews" has made it more or less usable again for now. But the writing is on the wall, I am not using GH for any new projects.

I believe they use semantic (Haskell program that uses tree-sitter) for navigation: https://github.com/github/semantic

So the answer may be that semantic does not yet have support for the language in question.

correct. adding support for new language is a matter of opening up a pull-request with the appropriate gluecode. https://github.com/github/semantic/blob/793a876ae45d38a6bd17...

It would be more realistic to think of pull requests if github allowed per-repo configs with custom parsing scripts, similar to gitLab LSIF support - https://docs.gitlab.com/ee/user/project/code_intelligence.ht.... Otherwise, how can a contrioutor be sure that his pull request really produces convenient code navigation in github UI. There is no way to test it.

The Semantic seems not used anymore:

> During this time, the team migrated from a Semantic-based tagging service to one that operated entirely with (newly developed) Tree-sitter queries. Though Semantic performed well, using the Tree-sitter query language allowed faster iteration and avoided the operational overhead of a program-analysis framework.

https://queue.acm.org/detail.cfm?id=3487022

Update: https://github.com/orgs/community/discussions/55704#discussi...

I believe this paper has an answer to your question - https://queue.acm.org/detail.cfm?id=3487022

Thank you very much for the link.

I don't see an answer in there, though.

They use tagging (as described also here: https://tree-sitter.github.io/tree-sitter/code-navigation-sy...). From both docs I assume `tree-sitter tags` works out of box for any language that has a parser. (Since neither doc instructs to save a custom config for tag extraction query, I assume every language plugin provides tagging queries).

I worked with tree sitter a bit for a small project. It seems to be the case that, while it covers a large set of languages, the actual implementation of each subset doesn't follow the exact same underlying API. And it follows that there is no universal AST parser as far as I can tell. Languages have many types of atomic parts to such an extent that unifying them is a tremendously complex task.

Have you used the `tree-sitter tags` command?

Unfortunately not. My stint of use was focused around dynamically loading in any given tree-sitter-langaugeName package via docker and then parsing any given file type in a runtime environment, in order to coalesce differing languages' parts into a unified metadata structure.

So, I wasn't trying to identify named things, like the tags command might seem to do, but more generic parts of the langauges like function, var, import/export statements, etc. I ultimately made my own walk functionality, but pulling out those atomic parts per language is still a very heavy task (basically need a mini lexer for each lang), even though TS seems to provide all the info to do the task.

ah, easy. it's because support has not been added into https://github.com/github/semantic which is the tech that powers the GitHub UI. Adding support is pretty easy/mainly glue code [1] that imports the tree sitter API.

[1] https://github.com/github/semantic/blob/793a876ae45d38a6bd17...

Reply on news.ycombinator.com