Lessons from building GitHub code search [video]
youtube.com
youtube.com
1. Press dot "."
2. CTRL + SHIFT + F
3. Enjoy results powered by rg (ripgrep)Grep is not as good, but the cost is like $500/year for infinite users vs $50m/year with a marginal cost for new users.
Also, my general comment on pricing applies here. Copilot is $100/year. So I’m supposed to think being able to search my code is the same price as being able to write the code.
Also, they are kind of in trouble because this will exactly be replaced once I just fork over $500/user/year for superchatgpt and it will definitely cover code search.
Sorry you've had a frustrating experience with the product. When we look into complaints about missing results, it's almost always one of two things. First, the repository is not indexed yet, but searching triggers indexing. Future searches work. Second, the files or the repo hit our documented limitations (https://docs.github.com/en/search-github/github-code-search/...). I think these limits are pretty reasonable, but we need to do a better job making it clear why content isn't in the index.
"One or more repositories of this search is not yet indexed, please try your search later for accurate results."
Likewise for the 2nd case.
I understand this is important. But the issue I have is that it’s hard or maybe impossible to know what’s been indexed and what isn’t.
I run a few orgs with hundreds of repos. Which are indexed? I don’t know.
This makes your search suck for my organization. I understand the reasons. They aren’t reasonable for me. I don’t want to search using your tool if it won’t work for my org.
Code search isn’t just for what’s popular. It needs to be for what is real and accurate.
We've been burnt sooo many times ('are you sure we changed all the occurrences of this in all our 3252 microservices?').
Between shoddy UX (the + being part of the code you were copying!), blatantly missing features for years (ZenHub, late GitHub actions), GitHub really succeeded only because of timing and network effect. The developers' community is quite powerful.
I'm happy for the original team, but now it's just another MS acquisition I hope we'll stop using.
Personally, my Firefox Nightly on Android will regularly crash (along with the whole System UI) when opening / trying to type into a GitHub page
There's a hanging draft for it in Gitea (inb4 yes also Forgejo).
https://github.com/go-gitea/gitea/pull/20311
Maybe this is the push I need to actually start using taskwarrior...
There were a couple GH discussion threads about the dreadful UI rollout, but someone complained one too many times about (code) search returning the wrong results so they've since been locked. For me it's the issue search that seems to consistently return the wrong results, and the text input widgets that are glitchy as hell in Firefox.
Performance-wise it's worth keeping in mind that Github is now rendering all the text client side. If you go through the discussion threads you can see where the GH devs didn't know how to account for multibyte codepoints and how that broke their non-native rendering with things like emoji and Chinese characters.
Stale data, PR’s and such don’t update properly, page loads are glacial, half the time it’s doing some loading bar of its own-no doubt as part of some trying-to-be-clever SPA thing, and it’s awful.
PS: where/what can I follow to know when it is published? Could you share your colleague github handle (I guess he'll have one :) )?
I know we don’t typically say this here but, you have a very relevant username for your question xoranth :D
Could you please give an update on whether or not GitHub is still considering adding “sort by recent” to search?
——
E: I just saw you answered that already. It’s a dearly missed feature.
GitHub’s previous search was not great, and when the new version launched it was a massive leap forward, where it’s now part of my daily workflow. Before this I thought good search at this scale might just be an intractable problem. :)
On the plus side I don't count how many reports I've done to companies who did leak not only their username/password but also all the cool proxy you could use to go inside their network. The weirdest one of them was a guy working in security at thales which is supposed to handle security sensitive stuff for governments leaking all that information as he was working on a side project involving poker during business hours ...
Pretty sure some people also made a fortune in crypto exchange API keys because I’ve seen threads where people advertise services to “cash out” Binance API keys for 5 cents in the dollar. I assume they use the balance in the account to inflate the price of some random coin that the attacker bought just before the attack. Yeah, that’s what this world is coming to.
https://www.videogist.co/videos/lessons-from-building-github...
• GitHub's previous code search was slow, limited, and did not support searching Forks due to indexing challenges. A new system called Blackbird was built from scratch to address these issues.
• Indexing code poses unique challenges compared to natural language documents, such as handling file changes in version control systems and deduplicating shared code across repositories.
• The talk discussed techniques used in Blackbird like trigram tokenization, delta compression, caching, and dynamic shard assignment to improve indexing speed and efficiency at scale.
• Architectural decisions like separating indexing from querying and using message queues helped Blackbird scale independently without competing for resources.
• Data structures like geometric XOR filters were developed to efficiently estimate differences between codebases and enable features like delta compression.
• Iteration speed was improved by making the system easier to change through frequent index version increments without migrations.
• Resource usage was optimized through techniques such as document deduplication, caching, and compaction to reduce indexing costs.
• Blackbird's design allowed it to efficiently support over 100 million code repositories while the previous system struggled at millions.
• Building custom solutions from scratch can be worthwhile when leveraging data structure to outperform generic tools for a domain.
• Anticipating and addressing scaling challenges at each magnitude is important to ensure a system remains performant as it grows over time.
Poor society we are part of, if everything needs to be consumed in 5 minute chunks.
One should also think and reflect about the content being presented. Grasp the ideas.
It is also about honoring the time the speaker put into the presentation preparing it.
Has there been a systemic fix for that issue, other than asking GH support teams to reindex the repo?
I'm currently building a query language whose grammar is very much inspired by Github's search syntax. I'm using Lezer, which is a GLR like Treesitter, so this talk learned me some parser generators (I've no formal CS education). Here's my grammar, a playground, and an example search query if anyone wants to play with it
https://github.com/AlexErrant/Pentive/blob/main/app/src/quer...
-(a) spider-man -a b -c -"(quote\"d) str" OR "l o l" OR a b c ((a "c") b) tag:what -deck:"x y"
First of all, the old system didn't really implement sorting for "recently changed" files. It implemented sorting for "recently indexed" documents. This was a decent proxy when we only rebuilt the index every two years. But as I discuss in the talk, we rebuild the index all the time now, so it would be pretty worthless as a sorting option. Another reason is due to the content deduplication I also discussed in the talk. When a blob is shared between repositories, what time do you store? Finally, it's complicated because of how Git works. If we did implement this, we'd like it to reflect when the path changed in the branch, which is what people mean by "when did this file last change". But Git doesn't have an efficient way to retrieve that information, so we'd have to build a system to keep track of it.
In short, it's hard to do right and easy to abuse. :-(
An alert on recently indexed content that matches keyword subscriptions, ala google alerts, would be an excellent alternative for that use-case.
I get this, but why break things if you don’t want to fix them? That’s great that you want to work on other things, but that feature was useful and existed for years and people depended on it. I pay for GitHub and you’re taking away features.
Not you personally, but this attitude is frustrating for me.
As a user, I don’t agree that the new features you implemented are better than the ones you took away.
It’s your choice, of course, but I don’t like this shift in dev mindset where really basic features that have been around since Unix time and are essential to programmers aren’t implemented because they are too hard.
Thanks for engaging in this thread and glad you’re working on this. But hoping since you’re involved in the development that you might be able to shift things a little toward “the good way.”
90+% of the time I'm executing a code search, I'm looking for example uses of a library API in open source code. But most of the time, Github code search just surfaces pages upon pages of the same exact code snippet across the origin repository and hundreds of forks.
For example, one search I executed earlier today: https://github.com/search?q=load%28%22%40rules_foreign_cc%2F...
https://grep.app/search?q=load%28%22%40rules_foreign_cc//for...
If you don't want to see forks, you can exclude them. Here's your same search, but converted to a regex and not including forks. I only get 3 results.
https://github.com/search?q=%2Fload%5C%28%22%40rules_foreign...
Again you could disagree but it is a decision that can be rooted in sound logic.
Or, maybe this is all value-destroying nonsense that only makes the service worse.
In any case, the performance of the new search system is supposed to be better, not worse. [1]
[1] https://github.blog/2023-05-08-github-code-search-is-general...
What makes you think that logged out users aren't adding any value?
The behaviour of github here is consistent with them assuming that logged out users provide a small but non-zero amount of value for them.
The link you provided seems to suggest that the code search is supposed to be faster for users, but I can't see anything in it that suggests that it's less taxing on the backend?
I'm curious because I'm not sure of the answer myself: what value do you think anonymous users provide to them?
I suspect gating it behind a login is a simple way to rate limit, and prevent abuse with a cheap and effective way to mitigate false positives. As someone who's had to make similar decisions in the past (albeit smaller scale), I can sympathise with them turning off a computationally expensive feature that's not at the core of their product, but just an add-on.
Also network effects of those users since GitHub is not just a git repo host but a social media and community for code.
I think product needs to strike the right balance of free and paid features to maximize their profits.
Some of the things you highlighted (ie the social network effects) are best experienced when logged in, and other than "just" browsing, there's not much in interaction for logged out users already (can't host code, open issues, comment, star, etc). I think allowing people to browse code and its history, alongside other parts of the platform for logged out users already feels pretty open to me
I think a service should focus on how it provides value to users, not the other way around. People use it because of what it provides to them, not because they desire to support the company's bottom line.
GitHub built a remarkably valuable product, which attracted 100M users and $1B in revenue. It has tons of public features that never required login. They could have put search, or any number of other things, behind a login at any point in the last 15 year. But they didn't, because that's not actually a good thing to do. Doing that can only ever make the feature less usable.
What drives them to do it now is unclear. Perhaps just wanting to juice their login numbers or whatever. But, performing little bait-and-switches on random features isn't going to make people use them more or spend more money. It's going to make them seek hedges against the destruction of the features they enjoy.
As for my link, the search result speed for users is directly related to cost per query, unless you believe that all of this is fake and the new search system is secretly just a bunch of expensive new hardware.
In principle they could make it all login only, but hosting open source repositories is one of the main charters of GitHub or at least it was. So it makes sense and is consistent with that vision to keep code browsing free without login. Not to mention the requirement to be available on search engines.
Code search however doesn’t seem like it’s some mandatory part of that mission, I mean all they’re asking is that you login. People in this thread seem to have a chip on their shoulder believing corporations should just give them compute for free. No they don’t!
If the goal is openness, i.e. why GitHub has the prestige it does, allying itself with open software development goals, then you should be able to do most things without having to sign up for the service. That includes browsing and searching. Neither of these need your identity.
On the other hand, if GitHub really just wants a walled garden with paying customers (either directly or the money to be made from datamining their identity and activity), it ought to shut itself off completely and get no benefit from being associated with openness.
What they've done is insidious. They're open, they're the trusted custodians of open source, but ah ah not really. For a preem experience you gotta pay up, choob! They let you search for 15 years but now they don't. And here you are cheering them rather than questioning them.
What's next? The certificate transparency project requires sign-up and logins, it's just too expensive to let anonymous users see the transparency logs you see, and by some amazing co-incidence, everyone signed in @google.com sees no results for mis-issued GMail certificates?
I guess that is why Microsoft acquired them.
1. Effectively serves as a walled garden for copilot to make sure that competitors can't use their data (your code) in coding assistants, knowledge discovery, etc.
2. Ensure that competitors who are using github are maximally mineable so they can easily discover and implement their solutions to eat their lunch (much like OpenAI's strategy here)
This is covered up by the excuse of performance, but it's fundamentally no different than the loginwalls we saw go up around twitter and reddit
... except this time done by a big industry player who LARPs as part of the open source community but is positioned to cannibalize it from within.
IMO It's much less about the legacy code (there exist already terabyte size datasets that take in a lot of things on github) and MUCH more about how up-to-date your LLM/AI is with new repos, "best practices" (or most common practices), etc.
Plus you often get "LLM Code poisoning" from older training data as it attempts to use functionality which has experienced breaking changes versus the current stable release. Current is King.
Also there's the whole goldmine of github discussions, issues, etc that a repo just.. doesn't have.
Right now you can still easily index those (though iirc they sometimes ban datacenter IPs), but they may also fall victim to the loginwall.
For reference, about 99.5% of the search queries that goes toward my search engine is demonstrably bot traffic (excluding the traffic that goes to the free and public API endpoint I offer).
A capable search functionality isn't exactly cheap to run, so requiring a login is the obvious way to be able to rate limit searches, which is otherwise impossible against a botnet.
Don't get me started about how it's harder than it needs to be to limit the search to your own repos