Koders: Search Over 3 Million Lines of Open Source Code Around the Web | Hacker News Reader

Koders: Search Over 3 Million Lines of Open Source Code Around the Web

koders.com

49 pointsdylangs103023 comments

Not ready for prime-time. Not even close. I wonder if they ever tested their query engine, it's hardly useful. No quotes? Queries ignore important characters like "_"? No roll-up of identical results? It also misses methods in its "(num) methods" results (or throws them into a totally different order) in several languages, and has some crazy match-ordering behavior.

An example: I searched for "helper_method" (a Rails method), and found these (bracketed what was bolded as the search hits):

  class.Scheduler.php:
  // Holds [method]s for ...
  Editor.py:
  from Models import Editor[Helper], Controllers
  AutShare.h:
  // IShare [method]s

and then I get over a dozen literally-identical files called "helper_method_example.rb" (tests with these results:

  require File.dirname(__FILE__) + '/spec_[helper]'
  
  describe "a context with [helper] a [method]" do
    def [helper_method]
  
  and the rest of the files contained this after those lines:
      "received call"
    end
    
    it "should make that method available to specs" do
      helper_method.should == "received call"
    end
  end

There are about three pages of results like that, or the same wrapped in a `module HelperMethodExample`, with identical contents otherwise.

It's been like that for years. Can't see it improving, now that they're going to have a monopoly on the market.

I will happily pay money to anybody who can build a code search service that is actually functional.

This isn't exactly the same thing, but I launched a search engine for programmers (that is, it doesn't ignore special characters) a few weeks ago: http://symbolhound.com/

It doesn't search code repositories, but rather tutorials and question/answer sites like stackoverflow. Right now stackoverflow dominates the results but that should change as we index more sites.

Would you be interested in paying per search, paying for access or just general donations based on your experience?

I ask as I am doing this but have not considered a business model for it as yet.

Micropayments suck, currently. If that ever changes, I'd be in favor of both per-search and access; until then, maybe a freemium setup? It'd be hard to hook programmers without letting them try it, and you can experiment with more computationally-expensive searches (regex, AST, etc) without letting random users tank your system.

I wonder if you could even build a market off open-sourcing the engine, thus allowing people to improve it for you, and run a paid service that searches GitHub, Google, etc for results. Free accounts for contributors that fix bugs or provide speed / feature improvements?

I dunno, just throwing ideas out while they're percolating, I haven't given them a whole lot of thought.

Yes I don't like micro-payments myself either. Freemium is my current plan.

I had also considered the GitHub model. I will be making portions free-software as the whole thing is built off it anyway.

If nothing else I will be going free for quite a while as I believe that there should be a code search repository available online.

For the record it will live at http://searchco.de/ I was in the middle of pushing out my second release when Google made its announcement so its looking a little worn at the moment.

Thank you for your interest and comments. The Koders team is in the process of re-implementing the code behind Koders.com. Please don’t draw too many conclusions about the future functionality of Koders by experimenting with the current one. Here’s some feedback on a few of the comments.

Why we are moving from .NET to Java technology is easily addressed: the .NET implementation was built by a company we acquired; we are a Java shop. We can invest in and expand the code search engine and site more effectively using technologies familiar to our developers.

We haven’t yet upgraded Koders to the latest software version. But we’ve done quite a lot of work beyond just a vanilla language port. For example, search results will be filterable to a single (or multiple) projects as mentioned in one of the comments and by language(s) as mentioned in another. We have noted the feedback regarding duplicate results, underscores, quoted searches, etc. and will put these in our backlog.

In preparation for the debut of the new Koders, we are currently doing scalability testing to ensure that performance of the re-launched site is up to par. We will index a lot more code than is currently there. Our test instance is now running with more than 7 billion LOCs in-house, about 2x the size of the current Koders.com index.

If you’d like to download and try the Java-based implementation that will be deployed to power the new Koders, it’s at http://www.blackducksoftware.com/java_implementation_downloa... .

We look forward to making this a great code search resource. Again, thank you for your interest!!

Searched for "i=i+1" and got many unrelated results. Tried "new Tree", got results for "Tree". Realised that quotes do not work. Not sure what use the search would be for me.

"this sucks" kinda works. :)

Even more problematic, I tried searching for zmq_init, to see if it would be useful for finding how other people had used a specific API function, and it seems like it treats an underscore just like a space. Not one result on the first page contains "zmq_init" in the summary, it just matched "zmq" by itself for all of them. I'm sure some of them do contain zmq_init in the complete sources, but that's not what it's matching, and for a less common API function, it seems likely a regular google search would be more productive than this.

Hopefully they can fix that, just having a bunch of source code indexed, or even available to be indexed if it turns out the current indexing is no good, is a good first step. But it seems like it needs work right now.

"We just finished migrating the entire code base from .NET to Java"

Interesting. I'd love to hear more about the rationale behind this move. I can understand moving from .NET to a dynamic or functional programming language, but why move to what is effectively a peer?

I'm not part of the team but a qualified guess would be the hosting and/or license fees.

And in either case, moving from .NET to Java would seem more rational than moving to a new functional language because they are so similar and wont require as much work. I wouldn't be surprised if there even are automatic converters.

Could it be that a company dedicated to promoting, analyzing, integrating, and categorizing open source software wanted to use an open source solution themselves? And although the open nature of Java can certainly be questioned due to the manipulations of Sun (and now Oracle), it's certainly more open that .Net, at least since 2007.

My guess is first-class support for libraries like Lucene and Hadoop, and to avoid software purchases or cloud surcharges.

Classic indicator of a soon to be failed project: "We didn't like language X anymore do we rewrote the whole thing in language Y"

There are always other "reasons", but they are usually just straw men. Users don't care what your site is written in.

I imagine this is on the front page due to Google code search going away. Since i'm currently working on a replacement this seems like as good a place as any to ask people what they are looking for. My list of required launch features are,

1. Regex search support. 2. Browse full file. 3. Filter search to single project.

This is based on what I observed on the Reddit and HN discussions when it was announced. I can see from the below some people would like grouping of identical results which is fairly easy to add so no worries there.

Any other thoughts?

> 1. Regex search support. 2. Browse full file. 3. Filter search to single project.

Regex search doesn't seem like a requirement here; for searching code, it usually suffices to include symbols when you index. Might also help to have some language-specific text classifiers, so you can say "search comments" or "search strings" or "search code".

Browsing the full file seems quite important to check the results. For that, you'll want line numbers, per-line anchors, and basic highlighting support for popular languages, in order of priority.

Filtering to a single project doesn't actually seem that helpful; if I know what project I want to search, I can clone it locally and grep it myself. Code search engines help with questions like "does anyone use this API?" or "how have other people processed the data from this library?", which need to search all projects. Filtering by language might help, though (to avoid problems with APIs provided in umpteen languages).

Index by symbols. ctags would be a fine start.

I have been wondering if that's actually required. Take for example the following search

print_r

During the indexing process it would take print and r as separate words. Any search for "print_r" should break up the same way, and assuming the search is phrase heavy return any document with "print_r" near the top anyway.

I can envision a relatively easy way to implement this without writing lots of code. How big do you think the market is for internet-based "code search"?

I think code search is very useful and I've come near to trying to construct a searchable database just for personal use. It's an itch I've been tempted to scratch. But it would require some time.

Big enough for several projects to already exist, koders (and a few others I have forgotten the name of), but not big enough for it to be worth maintaining for Google. No idea what number to put to that, but I would image its less then 5 million users.

So far I have mostly everything working, its just pulling down enough repo's to make it worthwhile.

I think you mean billion, not million.

How about this? http://searchco.de/ (I'm not affiliated)

Thanks for the link (I am the author). No code search in there yet, just API and snippets.

I will soon have code search as per Google Code Search. I just need to finish pulling down several thousand more repos before pushing it live.

For the moment however look at the following,

http://codase.com/ http://opensearch.krugle.org/

Reply on news.ycombinator.com