Towards Natural Language Semantic Code Search at GitHub | Hacker News Reader

Towards Natural Language Semantic Code Search at GitHub

githubengineering.com

159 pointsChris91152 comments

It is a real cultural problem how engineers get more excited about machine learning than basic usability.

GitHub search can't even search for a literal string, let alone a regex. It can't search a subdirectory. Ranking is indistinguishable from random. It's been this way for years. How about building an actual, usable, basic code search and then getting all fancy with your machine learning?

I almost built my own "online git grep for GitHub" last year.

I agree with this sentiment 100%. I can use traditional search engines for "how to ping a rest thing in python", but I can't grep Github for even basic snippets of code. I don't think their global code search has ever been useful. Glad they have their priorities straight /s

Agreed. Luckily, we as a community have tools like Sourcegraph which are based on battle-tested pragmatic systems from places like Google.

Disclaimer: no affiliation, just love the team and product.

At first I thought this would replace Sourcegraph, but looks like it's just an experiment with NLP... Thank goodness we have Sourcegraph for searching GH but especially for searching GHEnterprise in an SOA environment where it's impossible to have every repo cloned locally for ripgrep.

P.S. I'm not affiliated, we just use Sourcegraph at a company I work for.

Thanks for the recommendation

Damn, Sourcegraph is very close to something I've been thinking of building.

Join us! We are hiring and growing quickly.

All I can say is that we know this. We know it should be better. There's definitely more to come.

I'm building one! https://codesearch.aelve.com

Currently it runs on a fairly slow machine, so regex-heavy requests will take some time on big package repositories like Rubygems, but I plan to get a nicer machine soon.

If you know Scala, you can even contribute (wink wink), just ping me. A lot of tasks we have at this stage are pretty basic.

Nice! Next index all public Github repos?

You also can't search a forked repository, which is pathetic.

This limitation is especially frustrating when a fork becomes the "primary" repo for a project for some reason. It's probably not a common occurrence overall, but I've run into it at least a couple of times.

A good example is that GitHub's own repo for their CommonMark implementation isn't searchable, because it's a fork of cmark: https://github.com/github/cmark/

Exactly this. I don’t understand why the-thing-I-searched-for.java is so rarely on the first page of results. Doesn’t that seem like an obvious thing I might be interested in?!?

Yes! When I read the post title I was really excited. The I clicked in and felt my heart sink a little. Engineers and PMs seem to be too easily swayed by shiny things.

To be fair its a hard problem to solve, especially with traditional search engine tools.

Take for example,

    for(int i=0;i<100;i++)

And then a search for i++ Due to the way almost every search tool works that would be split into tokens "for int i 0 100" which are not very useful. Even if you include the characters = ; < + ( ) in the search you break the ability to do things such as boolean queries or fuzzy search term~1

Its totally possible to solve these issues using tweaks of the input into your index, which is what I did with searchcode.com or with a different approach which is what Google Code Search did. However neither have a requirement to be 100% in sync with the repository which I suspect is something that the github team value.

All the code search tools suffer from this in some way. At small scale its possible to just brute force the search. At scale you can do it by tweaking your algorithm and sacrificing accuracy. My feeling is that the github team chose accuracy.

But people use grep on their code all the time ...

People use grep on their local code repository, which is generally less than 2 gigabytes of source. A tool like ripgrep can process that in under a second on any modern machine with a warm disk cache.

Its when you get to hundreds of repositories or 10's of gigabytes of code that local tools cannot run fast enough. They are not designed for this use-case, and usually rely on the files being searched hitting the disk cache for repeated search performance.

It may be possible for github to shell out to grep for a single repository search (I have no idea how the back-end works but I doubt its impossible) but, I suspect that almost everyone wants/expects this to work across multiple repositories or across all of the github repositories.

Since its not easily possible to do so across everything they are not adding it to even a single repository to avoid search working differently for different situations, which is a fair approach in my opinion.

Nobody is asking for cross-repo search. Literally just let us run "git grep" on Github.

On your local machine, you can get away with a linear scan (as "git grep" does), because nothing else is contending for time with your search. "git grep" is actually very costly in terms of CPU and disk IO, but you don't notice, because you're only running one "git grep" at once, and so nothing is contending for those resources.

On GitHub, tens of thousands of searches will be happening at once on the same search cluster. If they were all literally doing a "git grep" (a linear scan of the associated repo data), the disk caches would thrash back and forth between queries, and nothing could be answered in less than 30 seconds.

The only way for GitHub to respond to code searches at scale in a reasonable time, is to have a pre-built index.

If the index was per repo, that'd be a kind of partitioned index; and there's no DBMS that I know of that can handle a partitioned object having 58 million partitions. It makes much more sense to have an unpartitioned index... which effectively implies "cross-repo search" (because, once you have an unpartitioned index built up over all your repos, it costs nothing to enable someone to search that entire index at once.)

While that's basically true, git grep doesn't scale...

With ever-cheaper RAM, ever-faster SSDs, and cloud computing, it could actually make sense, now or soon, to scan through an entire repo, either on "disk" or in RAM. I started building an app that would suck down any GitHub repo by simulating a git clone in the back-end, in the time it took you to type in your query, and hold it in RAM across queries. As a user, I'm sure I would gladly pay however many cents it costs to take up this much RAM when I'm on the search page over the course of a month.

Even if a linear scan isn't feasible, regex search at scale is not an unsolved problem, and GitHub has access to world-class engineers. Google Code used trigrams to do regex search with an index (https://swtch.com/~rsc/regexp/regexp4.html). Sourcegraph offers regex search.

When I press "download" on GitHub, it also does a linear scan ...

Of the files, maybe, but not the characters in those files. It's orders of magnitude more difficult.

it would only really make sense to have one big index, and to narrow the search to particular project in that index. This of course has the benefit that you can move out to the whole index if you want.

You could then offer code search across the whole index (minus any project marked private that you are not a part of) as a paid offering.

In full agreement. Every now and then I'll expect a search to work. My solution has been to run etsy/hound [0] for my active reps.

  [0] https://github.com/etsy/hound

Could not agree more. Everyone who works for or have worked for Google in the last years knows that an excellent code search does not have to be fancy.

I've honestly moved to Google, adding

`{search query} -site:github.com/{repo}/{file i want to target}`

Its much clearer and concise.

I made a regex search for GitHub and emacs plugin. In theory I could put this on GitHub. It uses the bigquery ghtorrent table. There's only so much time in a day though. If you want it upvote me

So what? Here their only mistake is not to license/buy some search engine instead of wasting years on developing another "meh" one. What they are doing here with semantic search is the future and their chance to make all existing code search engines obsolete. Use your favorite Internet search engine to find GitHub's snippets of code instead. Those won't give you semantic code search though.

This might just be me, but does anyone else feel that GitHub's code search has other points that could be improved first?

My biggest gripe is that the other results show in seems to be totally random. For example, if I have a Java class called A and I search "class A" in code search, the actual A.java doesn't tend to show up anywhere near the front. I just tried this in a repo and the actual A.java file was on the last page of results when I searched "class A". The vast majority of the results before it didn't even have the words "class" and "A" next to each other, which A.java does...

Maybe I'm doing something wrong (I'd welcome any input on how to use code search correctly!), but it just feels like they're jumping the gun on trying to make their code search more advanced when the basic functionality doesn't work that well.

Yes, GH search leaves so much to be desired. And this post doesn't actually seem to address the weaknesses.

The search appears to be configured for natural language documents, not code. The stopwords are not right and search appears to strip all sigils. They could get pretty far just by parsing documents and changing their lucene/elasticsearch configuration.

We are very aware of the problem. I think you are going to really love what we are working on.

Is it some search algorithm that is so new, unusual and groundbreaking you can't talk about until it patented, or what?

I really hope you are right and have your priorities straight when it comes to search. I'd love for a way to search for usages of a class::method, or for strings that contain the text "hello" or for variables named foo. And if you integrate that into the code itself, Ctrl+click a class method to find all usages, maybe even usages in other repositories, so I can see how other people use a certain library.

And of course good old regex search.

Sourcegraph does this quite well if you need.

They are tagging so much code with text, that i'm sure they have something groundbreaking.

I would settle for the ability to use logical OR when searching issues/pull requests, or to combine multiple negated searches.

"is:pr is:open ( author:bob OR author:jim )"

The lack of this pretty basic functionality makes issue & PR search much less useful than it could be.

Agreed. It'd also be nice to see a list of issues you're subscribed to. Here's a fun issue to follow for that - https://github.com/isaacs/github/issues/283

It is awesome that they are working on this, but can I just say there are a lot of basic search features they need to add before "doing the hard thing". Here are some things that I should be able to do easily but can't (or can't very easily or well) using GitHub's search mechanism:

1. exact or close string searches for code that involves ![]{}_-*() etc characters

2. searches across past commits (e.g. find a line that used to be in the code)

4. search across pull request + comments (not just issues and commit messages)

5. advanced search operators -- there should be a full filtering UI with ands and ors etc

Because of this I often find my self grepping locally, or (more often) totally out of luck.

Now that’s what I call a misfeature!

GitHub is used by programmers. Surprisingly, they tend to be very good at telling computers precisely what they want, in the computers’ own language.

Natural language search is the exact opposite of this, invented for mom & pops who start their search phrase with “Dear Google, I’d like to search for ...”.

GitHub is building some amazing stuff recently, I guess now that Microsoft is going to acquire them, there's far less pressure on making Github Enterprise profitable..

I saw this created in another thread and it seems to accurately sum up the comments here: https://imgflip.com/i/2i90x2

What thread did you originally see that in?

They should add regex search support first before this stuff.

wait, they can't search through forks or collate identical results and they are going into natural language processing?

Devs don't search code repositories using natural language queries, and any scenarios of searching for code examples that way are already extremely well handled by StackOverflow and Google.

This is an incredible waste of time and resources that could be spent making the existing search far better with very minor tweaks. A perfect example of big company project management where nobody seems to know what their users actually want.

I'd settle for github search that's case sensitive and recognizes things like dollar signs, semi-colons, commas, braces, and such.

Dear GitHub,

Please build search that lets me actually find a given file by name.

You are busy building a space rocket when all we want is a bicycle. Impressive, but useless for just popping down to the shops.

Love,

The rest of the world's developers

I want to work at github. They're making cool things.

As of today, GitHub has 89 open positions:

• 2 openings - Business Systems

• 2 openings - Communications

• 38 openings - Engineering

• 3 openings - Finance

• 1 opening - Internal Communications

• 4 openings - Legal

• 8 openings - Marketing

• 2 openings - People Operations

• 1 opening - Policy

• 7 openings - Product

• 8 openings - Sales

• 9 openings - Security

• 1 opening - Services

• 3 openings - Support

— https://github.com/about/careers

Do they? The main product appears to have 0 product velocity

I used to feel this way, but then I discovered https://blog.github.com/ and no longer do.

Sure, they may not be addressing your/my specific concerns, but the product is changing.

I want a MS.

Reply on news.ycombinator.com