GitHub code search – waiting list signup | Hacker News Reader

GitHub code search – waiting list signup

github.com

99 pointsmmq48 comments

Hello from the GitHub code search team! I'm really proud of this release. It's the result of a year of hard work from multiple teams to bring the custom code search engine we built for cs.github.com to github.com. In the background, we've also spent a lot of time since the launch rebuilding the index format to let us scale up the system.

My colleague Tim Clem will be speaking at GitHub Universe today about some of technical details: https://githubuniverse.com/events/detail/virtual-schedule/49...

Can you search for special characters, and are large files indexed?

All characters are indexed (UTF-8 only). Files larger than 350 KiB are currently excluded, though we welcome feedback on use cases that require larger files to be indexed.

A few weeks earlier and I could have not be forced to hack together a tool to search for specific repos in specific languages that contain certain code-snippets by scraping the existing search API for a week. :P

Joking aside though, looks really great, thank you so much! :D

Last time when I heard this, the GitHub search was the shittiest search ever I came across. It didn't had even basic dedup. You search for something and there will be 10 pages of results with exact same code from thousand forks. The intern guy who worked on it claimed he had used "AI" however.

Looks as non-promising as before: https://cs.github.com/?scopeName=All+repos&scope=&q=gpt2+tra...

Congrats on the launch Luke =)

Thanks! Nothing like a little friendly competition... :)

100% =)

I’ve been in the beta (I guess the alpha) for a little over a year now. It’s absolutely fantastic and has been great to understand, discover, and learn new code.

Great work!

Does this have any relation to https://grep.app/ ?

No, it is a custom-built search engine for code. grep.app is great, though.

One feature for code search that I've wanted for some time that could unlock some really interesting results would be to allow search to reach into the git file history.

I've found myself sometimes thinking that something existed in a particular codebase and I go to look for only to discover that I'm unable find it. Then, I realize that the code I was looking for has been removed. I can only find it by browsing back into the history figuring out the events that led to the deletion or change. Being able to search the history would be killer IMO.

I’ve run into this exact situation multiple times. Searching the whole history would be revolutionary.

FWIW you can do this with Sourcegraph today, searching over both diffs (code changes)[0] and commit messages[1]:

[0] https://sourcegraph.com/search?q=context:global+repo:%5Egith...

[1] https://sourcegraph.com/search?q=context:global+repo:%5Egith...

(I work there, just commenting on my own though. we're all pretty happy to have competition, more awareness of code search, etc.)

Are you familiar with git pickaxe?

For example: git log -pS 'deletedvar'

Thank you for sharing that! TIL.

This is something that I'm working on and I believe a couple of others as well. My solution won't be available until early next year though, as indexing public repos at scale is far from trivial and I'm expecting this will consume over 40TB of fast storage to be performant.

A couple of years ago you stopped indexing repos that hadn't had activity for a year [0]. Will such repos be indexed by the new code search?

[0]: https://github.blog/changelog/2020-12-17-changes-to-code-sea...

Our ultimate goal is to surpass the current code search index. As of today, there are about 25 million repositories in the new index, but we have more work to do before we reach full scale.

My only problem with the new code search is I've noticed some examples where exact string searches fail to locate some matches. I was looking for callers of a function and assumed no results meant no callers. But a simple 'search all' in VS Code found results missing from GitHub code search.

I used this recently when I was searching for examples of CPython extensions implementing a particular feature, and it actually did help me find what I was looking for! I also used it when looking for examples of Github action workflows that did particular things. It's nice that you can use the path: option to specify the file extension and/or the path to the file.

This looks very similar to the internal code search product at Google (called CodeSearch). If the quality of this is going to be anywhere near that of CodeSearch, I would use it every day

Honestly, I have caught myself looking for an advanced search option on my github page more than a couple of times. Just to remember that it does not exist. Long overdue!

Press . in any repo and then search :)

Does it still only index the primary/default branch of a repository, or can it search within all branches? If it can now, can we filter the search for stuff within the branches of forks for a specific repository?

So many interesting work can only be found manually, laboriously (but is so rewarding) within the explore network graph where people work on their own branches in their own forks. Their code is never found from the search function.

Currently, we index the head of the default branch. However, the index format will support branch indexing, and we use that functionality to index forks, which was not possible in prior incarnations of GitHub code search. Indexing specific commits (e.g., associated with a PR or feature branch) is on our roadmap.

https://docs.sourcegraph.com/getting-started/github-vs-sourc... highlights just such a comparison, and they have both "fork:yes" and "fork:only" and their "rev:" keyword supports branch globs

> Way more than grep.

Heck, it'd be nice if the current system right now at least did grep with users being able to toggle POSIX options. It doesn't even do that.

Instead, I have to clone the entire repo to do a trivial lookup. To be fair, supporting this would require GitHub to do the compute themselves, but I have no idea what they could possibly be doing right now that's more effective.

Inching closer and closer to each Github project page being one-click away from a complete, web-based IDE.

Is this a Sourcegraph killer or does SG have other features that still keep it competitive?

We (Sourcegraph) have other features to stay competitive. One thing we have been working on indexing other code hosts like Eclipse Foundation, Crates.io, etc.

We put this together anticipating these questions: https://docs.sourcegraph.com/getting-started/github-vs-sourc...

We also just launched Search Ranking and Relevance: https://about.sourcegraph.com/blog/new-search-ranking

More to come! And BTW we are happy to have competition, GitHub is a great brand and it will only make code search better for all developers.

Thanks for the response and for having a doc ready. Code search definitely seems to be on the rise as a needed tool

You're welcome. We agree, good times ahead =)

Is this the product that tree-sitter-graph [0] is being used for?

[0] https://github.com/tree-sitter/tree-sitter-graph

Yes, Stack Graphs power code navigation for supported languages (currently, Python). We also use tree-sitter parsers for CTags-style symbol navigation.

I have been using cs.github.com religiously since it originally went into beta and it has increased productivity for me to insane levels. There are many features I still want but its usually *good enough

Developers, developers developers! Very happy Microsoft bought Github.

This will also improve the github.com code search, right? I always found it frustrating that I couldn't really search for exact strings, even using quotes. I hope this fixes that too.

This will become the new github.com code search. Last year, we launched a Technology Preview of our new code search as a standalone site: cs.github.com. Today, we're bringing that powerful code search, plus a redesigned code navigation UX, to github.com.

I've been using this since January and it's become a near daily tool for me. Very good stuff

It's in beta. Will this service have a free version for individuals and organizations?

This is addressed in the FAQ. It will be included with GitHub, just like the current code search.

Digging it. Fingers crossed github's app search options are updated as well

Dark theme is the default theme now for modern web. I found it a good thing.

HUGE

Does it dedup?

Content is deduplicated, yes.

Reply on news.ycombinator.com