Hacker News Reader

Comment by ezekg | Hacker News Reader

I use their new code search a lot to grok how people use certain features, or implement certain things. But I do wish there was a way to filter out forks. Sometimes I search a string and just get a bunch of forks all with the same result. For example, searching a common class in a Rails app often just shows a bunch of rails/rails forks, which is a lot of noise to sift through when you're trying to see how devs commonly use a certain feature.

View on HN

Thanks for the feedback! That's coming, we've been prioritizing scaling the index and ingest process and haven't had a chance got add that yet. There are a bunch of value-add features like this I am looking forward to knocking out soon.

Well done

thanks so much, you guys are doing amazing work

Out of interest, if I have a repo with many millions of files that compress quite nicely down to about a 1.4gb packfile, is it better for the ingestion and/or indexer if I break this down into many smaller pushes or one large push?

Because I pushed such a repo yesterday and it’s still not been indexed.

Sorry, there are size limits on the number of files in a repositories that we'll index. That is probably why your repository wasn't indexed. Out of curiosity, what are you storing in this repository?

Sequences of code from Python package releases from pypi, as an experiment I’m working on. They compress quite nicely as the deltas between releases are fairly small.

I thought it would be nice to be able to search through them, but I guess the file limit was reached. That’s a shame.

Oooh that sounds really useful, would you mind dropping a link?

Sorry for the cyber-stalking orf, but I'm guessing it's this: https://github.com/orf/pypi-data

It's not - it's something that builds upon that. It's not really ready for prime time yet, but it will be here: https://github.com/orf/pypi-code-repo

if it works, which I really hope it does, we should have a repo containing every file (sub-25mb) published to pypi.

it's not really useful to clone this, but the git packfile + index seems to be pretty ideal for storing + querying this data, reducing 14tb of compressed packages down to like 100gb or so.

this should enable large scale exploration of the contents of pypi from only your laptop, which is useful if you want to look at how the python language is evolving (how many packages use f-strings over time?).

the rust tool i've built to do the heavy lifting is here: https://github.com/orf/pypi-import-test. i'm learning a lot about git internals, it's quite fun

Is it possible for you to clarify the bigram weight function? I'm assuming it's inverse frequency but that would a be great tidbit to better understand why the covering ngrams work.

As an open source library maintainer I've been using it for that too - it's fantastic for answering questions like "is anyone using this API method I wrote?" and "how much of a mess would it likely cause people if I deprecated this function?"

Would be nice to make a tool automating this and printing the "coverage" statistics.

I run into this a lot where I'd like to see some real world use cases of a method in FooClass, but get a hundred pages of forks of the FooClass.h header file. In some cases I've been successful adding "-filename:FooClass.h" to filter out that header file. If the method is used a lot in the primary project it can be a game of whack-a-mole but it often eventually works.

Sourcegraph[1] does this better and has done for a couple of years now. I use it for this reason all the time.

[1] https://sourcegraph.com

Yes, just change the URL from https://github.com/foo/bar to https://sourcegraph.com/github.com/foo/bar to be dropped in to a code search for that GH repo.

Perhaps it could benefit from something like a "dissimilarity" filter, which ranks the current result set by returning the most unique hits first. You wouldn't always want this, because sometimes you're searching for how something is typically used, and with many duplicative results you can confirm that's the preferred pattern. But other times you're looking for more estoteric usage of a certain function, and it would be nice to filter the "standard usage" from the results (although you can already do this with carefully chosen negated keywords).

Personally, I'm happy with the new code search so far. I stopped using Sourcegraph because I could never get the deeper results I wanted - it would just return the top five repositories including some common code snippet and I couldn't explore further than that. GitHub Code Search doesn't seem to have this problem to such a degree, since I can use negation more naturally, and since my query is not limited to some shallow subset of the corpus before refining it.

Re: Sourcegraph, we're working on improving that, and sorry you couldn't get the results you wanted. We primarily build for the code within customers, where this particular problem is less common than across all open-source repositories. But we want it to work really well in every case.

Our new ranking (https://about.sourcegraph.com/blog/new-search-ranking) should help a lot here, and it's live on https://sourcegraph.com. Can you share some of the queries you tried so we can see how much ranking helps and how to handle them better?

Sure! I will try it next time I'm using code search. I can't remember the specific queries I've tried, but I only ever used it when looking for "how have people used this function," so I remember the limited depth of results being particularly annoying.

Thanks for the link to your blog post. I didn't realize Steve Yegge had joined your team - congratulations, that's quite an endorsement!

I've always liked Sourcegraph just because our company names are so similar (founder of Splitgraph here)... we might have even gotten a few candidates because of that initial confusion. :)

How has GitHub Code Search impacted your product direction? Do you see it as an opportunity to focus more on the internal use case, or do you have plans for some other differentiation? It's always unfortunate when a big company introduces a product so similar to the core product of a startup, but I'm sure there is a silver lining there, especially when you have a talented team and a mature codebase (for example, Fly.io has been able to carve out a niche for itself despite Cloudflare moving to compete in the same areas). Either way, best of luck to you from a fellow S-grapher!

Similar to this, there are some generated files it would be nice to rank very low, or even completely ignore, as they probably weren't intended to be committed in the first place. Some search results are dominated by files in paths like ".npm/_cacache/...", ".apm/...", ".svn/...".

I think forks without any changes shouldn't really show up in search tbh.

When trying out some C++ I was surprised by how many projects bundled the library source code, essentially finding any actual client code a pain.

As a C++ developer, I maybe-ironically have this same complaint with JavaScript projects ;P. I particularly hate it when people embed minified versions of libraries, as line-based search and display then will experience a hit for essentially every query on the one giant line that is tens of kilocharacters long.

If your language lacks a de facto way to install dependencies, people will commit vendored code. See: C, C++.

If your language's package manager puts your deps within the repo directory by default, people will commit the vendored code. See: Node, Go (since go.mod).

As long as we can all agree that committing compiled code is a crime punishable by 24 hours in the shame cube.

I have this issue with GitHub code search very often. I'll search up a function to see some examples of how to use it, and the first 20 or so pages of results are just copies of the source where its defined.

I’m glad to see some folks in the comments section here using this. My first thought on seeing the headline was, “why not just use grep?”, which the article addressed. My second was, “who in earth wants to search all of GitHub?” I’m happy someone does.

But also, you know what you’re looking for and you can filter the results.

Someone less experienced than you may take any of those results as gospel. Gospel the same way StackOverflow could be, but this time by a source who will do it’s best to tell you what you want.

my personal rule about AI is about the same as the rule about being a lawyer. Never ask a question you don’t know the answer to.

Reply on news.ycombinator.com