Ask HN: Good source code search engines? | Hacker News Reader

Ask HN: Good source code search engines?

20 pointssupersan13 comments

I recently saw an article on HN about how to use GitHub's search as a starting point when writing code. That indeed is a great trick and it has already helped me save a lot of time.

Unfortunately, GitHub's search is very lacking and most of the time instead of showing relevant search results it just shows lots and lots of duplicates for the main library.

For example, I was searching for this[1], but the search results show the Main SDK files and not the files created with the SDK (which is what would've helped me). I couldn't find a way to suppress the files or duplicate searches.

So I tried looking for a good source code search engine and I've find none so far. I've tried almost everything I could find on Google and Quora but most of these services are either discontinued (Google code, etc) or very bad.

So, I was wondering what you guys use to search source code? And are there any other tricks like GitHub's search that can increase coding productivity (I think searching SO is also very useful).

[1] https://github.com/search?l=php&q=createEnvironment+ElasticBeanstalkClient&ref=searchresults&type=Code&utf8=%E2%9C%93

https://searchcode.com http://www.krugle.com

Thanks, I've already tried both of them and they're not returning a lot of results, esp. not many relevant results.

But so far I think searchcode is okay because at least it does show something.

Hi, I'm the guy behind searchcode.com Could you possible let me know what results aren't working for you? Im well aware that its due for some improvements in the search area and feedback is always helpful.

First of all great work! Of all the search engines I tried (and I've tried at least 10), I found yours to be the next best thing after GitHub search.

Anyway, here are some links you can try for yourself:

No php results:

1. https://searchcode.com/?q=CORSConfiguration+CORSRules 1. https://github.com/search?l=php&q=CORSConfiguration+CORSRule...

Only SDK libraries, what I think would be useful if the search engine somehow prioritized code built on SDK/API over libraries (though I don't have the first clue how you would distinguish the two but that would be a killer feature because that is super useful)

2. https://searchcode.com/?q=createEnvironment+ElasticBeanstalk... 2. https://github.com/search?l=php&q=createEnvironment+ElasticB...

I can provide you with more examples, if that helps. Thanks!

Thank you.

That's one that comes up a lot. The trick is knowing the difference between the two which I am still thinking about how to implement.

If you could email me at bboyte01@gmail.com that would be very helpful. It looks like most of these are down to just not having enough code indexed.

For the record I tend to crawl bitbucket/gitlab more often then github simply because they do not have a search and I want that code to be available. I have started a full refresh of everything though so hopefully your results should improve in a few weeks.

That is really great news. One way I can think off to distinguish a library from other code is how many times the file shows up in your search results, i.e. the more the duplicates of a file you find (in various repos), the higher the chance it is a library.

Thats a pretty good point I suppose. I do track that information so it wouldn't be too difficult to implement I guess.

Off topic but how would you build something like this from scratch. Just for the learning part of it. Any examples?

I have written about it all here (I maintain searchcode.com ) http://www.boyter.org/category/searchcode/ specifically http://www.boyter.org/2013/02/why-code-search-is-difficult/ is of note.

If you want to see implementations I suggest looking at https://github.com/OpenGrok/OpenGrok https://github.com/ggreer/the_silver_searcher https://github.com/google/codesearch and my own https://github.com/boyter/searchcode-server

Yes, that is very interesting to me as well. I think Amazon's ElasticSearch can be used for searching. GitHub would be the obvious choice for crawling the source code.

Of course, there are a lot of roadblocks, like how much GitHub allows you to crawl the source code (sometimes you can find huge dumps on legit torrent sites or I don't remember the name of the site but it provides you with S3 buckets with crawled data where the requester has to pay for bandwidth but the data is free).

Then of course you will have to index the source code by language, project, author, date, etc. It's definitely not easy, which could be the reason why there is this huge gap.

I've recently ran into debian code search again, https://codesearch.debian.net/. Built on top of Russ Cox's Code search (https://github.com/google/codesearch), and quite well documented.

Can't say I've used it much, but it gives a few pointers into how you could build a better one yourself.

There's a startup solving this! https://sourcegraph.com

Tried it, didn't return anything. Also for the life of me, I can't figure out a way for it to search php code.

Reply on news.ycombinator.com