Hacker News Reader

Comment by kjuulh | Hacker News Reader

I really like the new search. Though sometimes it is a bit deceptive. I.e. when searching for a function name by clicking on a piece of code and suddenly you are in an entitely different code base with an unrelated function though it shares the name.

It feels like github code browsing is a step between a full editor with lsp and a static site. I Hope they work out the Kinks and make it more smooth

View on HN

It sounds like you are navigating in a language where we currently only support "fuzzy" or "search-based" Code Navigation, which only uses the _unqualified_ symbol name as the search target. As you point out, this can be very imprecise when there are particular symbol names that are used a lot. (Think `render` in a Rails codebase, for instance.)

We also have Precise Code Navigation — still currently only for Python [1], but we're working on other languages as fast as we can. As mentioned down-thread, that is built on our new stack graphs framework [2]. We're often asked why we're not leaning on something LSP-shaped for this. I go into a built of detail about that in a FOSDEM talk from a couple years back [3], and also in the Related Work section of a paper that we just published [4].

[1] https://docs.github.com/en/repositories/working-with-files/u...

[2] https://dcreager.net/talks/stack-graphs/

[3] https://dcreager.net/talks/2020-fosdem/

[4] https://dcreager.net/publications/016-stack-graphs/

Supporting jump-to-definition natively seems like something that will be table stakes for any code hosting site in the future.

I can only think of one hosted repo service that provides a working go-to-definition feature and it is not github or sourcegraph. What makes you think this will suddenly become widespread? Github spent years doing this and their new thing is strictly non-semantic. It doesn't have the faintest idea where the name is defined, or if there's even a difference between a function name, a parameter name, or a word in a comment.

FWIW Sourcegraph has fully precise/semantic go-to-definition, find-references, etc. We use SCIP code indexers (a spiritual successor to LSIF, the Microsoft standard for indexing LSP servers)

Not for C++. To test my recollection I navigated to abseil-cpp/strings/str_split.h, clicked on the declaration of absl::ByString::Find, and clicked "Go to definition". I was presented with every function in Abseil named "Find" regardless of its scope or parameter types. That's not "precise code intelligence"!

In the top right corner of the tooltip it will say either "Search-based" or "Precise" - in this case, you're right, we don't have the abseil-cpp repo indexed so it falls back to search-based as you describe.

We do have a C++ code indexer in beta, https://github.com/sourcegraph/lsif-clang - it is based on clang but C++ indexing is notably harder to do automatically/without-setup due to the varying build systems that need to be understood in order to invoke the compiler.

(I work on C++ indexing at Sourcegraph.)

As my colleague mentioned in a sibling comment, we have an existing indexer lsif-clang which supports C++. I just added a Chromium example to the lsif-clang README right now: (direct link) https://sourcegraph.com/github.com/chromium/chromium@cab0660...

We are also actively working on a new SCIP indexer which should support features like cross-repo references in the future. https://github.com/sourcegraph/scip-clang

Right now, Abseil doesn't have precise code navigation because no one has uploaded an index for it. In an ideal world, we would automatically have precise indexes for all the C++ code on Sourcegraph, but that's a hard problem because of the large variety in build systems, build configurations, and system dependencies that are often specified outside the build system.

The rise and popularity of LSP and projects such as treesitter are a superb foundation for features such as this. Both support a wealth of languages, it is still and will be quite hard to assume the toolchain and settings required for producing accurate information though.

But this could be tied into CI, especially for projects utilizing runners for building the code.

So the barrier to entry now is orders of magnitude less and with github and others pushing for codespaces it could be the final piece to tie everything together.

> But this could be tied into CI, especially for projects utilizing runners for building the code.

One often overlooked drawback of generating this data during CI is that you, the project owner, are now paying for the compute. Of course, maybe you qualify for a free tier. And if not, if you're already running a CI job for testing/linting/etc, the marginal cost of also generating code nav symbols might not be too bad. But one benefit of the stack graphs approach mentioned up-thread (and the code search indexing described in OP) is that all of the analysis and extraction work is done in dedicated server-side jobs that GitHub is footing the bill for.

By necessity I already do it many times a day on my workstation. It would be neat for the cloud to start utilizing the endless compute available on client machines. A back-channel when doing a git push to publish some related artifacts?

> It doesn't have the faintest idea where the name is defined, or if there's even a difference between a function name, a parameter name, or a word in a comment.

I don't think what you are saying is actually true for stack-graphs[0][1].

[0]: https://github.com/github/stack-graphs

[1]: https://github.blog/2021-12-09-introducing-stack-graphs/

I can think of zero. Doing go-to-definition statically is very difficult.

I think that there should be some sort of standardized, language-agnostic metadata format for semantically indexing a codebase. It could include e.g type information for expressions or declared variables (for languages that infer types), and an index of symbols and how they're connected.

This metadata file would be generated by a language-specific tool. For instance, Cargo could generate it for Rust projects, ctags/cmake for C / C++, Sorbet for Ruby.

Then a service like Github / Gitlab / your own homegrown code viewer could provide things like "Show type", "Jump to source" without ever needing to build a language-specific parser or interpreter (which seems arbitrarily difficult, most build systems provide escape hatches, so you can't assume much about project structure).

Basically, LSP as a static file in a standard format that tools can read to understand a codebase, without needing to model the language's semantics.

SemanticDB (https://scalameta.org/docs/semanticdb/guide.html) is a protobuf-based file format that does almost exactly this for JVM languages, primarily Scala (I was a contributor a while back). It is used to build an intelligent online code browser, as the backend for a language server, and to do intelligent refactorings.

I think a language-agnostic semantic metadata format is a good idea, but requires a lot of compromise. ctags partially does this, but only to a very coarse level (mostly definitions and references). I think some ctags implementations also define 'extension fields' that could be used to give type information, but I don't know how/if these are used in practice. SemanticDB is extremely fine-grained, but highly specialized to JVM languages and type systems that are designed to work with the JVM. Finding a common set of semantic features that can be used across languages and type systems that is fine-grained enough to be more useful than ctags sounds very difficult to me.

I think simple things like "go to reference" or "show type" would be sufficient for 95% of usecases. But if you split languages up into a few different categories (maybe along the lines of Algol-like vs Lisp-like), and were flexible with extensions, I'd imagine we'd see some common patterns emerging, and clients would take advantage of that. Best effort is probably good enough to greatly improve the ergonomics of search.

This is pretty much exactly what we've built at Sourcegraph. Microsoft had introduced (but pretty much abandoned before it even started) LSIF, a static index format for LSP servers which encodes in detail all possible LSP requests/responses, effectively.

We took that torch and carried it forward, building the spiritual successor called SCIP[0]. It's language agnostic, we have indexers for quite a few languages already, and we genuinely intend for it to be vendor neutral / a proper OSS project[1].

[0] https://about.sourcegraph.com/blog/announcing-scip

[1] https://github.com/sourcegraph/scip

This is pretty much exactly what I'd imagined, very cool. Is the plan to open source all SCIP indexers?

All of our SCIP indexers are open-source: scip-java (for Java, Kotlin and Scala), scip-typescript (for TypeScript and JavaScript), scip-python, scip-ruby, scip-go and scip-clang (for C and C++).

There are also some community-maintained OSS SCIP indexers. - rust-analyzer: https://sourcegraph.com/github.com/rust-lang/rust-analyzer/-... - scip-zig: https://github.com/zigtools/scip-zig

Reply on news.ycombinator.com