Ask HN: Google not completely indexing github.com issues? | Hacker News Reader

Ask HN: Google not completely indexing github.com issues?

56 pointsjve34 comments

Given https://github.com/AzureAD/microsoft-authentication-library-for-dotnet/issues/3033

Fore query "cannot persist Microsoft authentication token cache securely!" (with quotes) Google returns single result, written in Chinese. Luckily I opened that Chinese result and spotted link to that issue.

duck.com does find the issue at hand.

I mean GitHub is no small site, but somehow I expected that Google will find my ANY public string on the internet.

Not that it doesn't find issues at all - but I stumbled upon this one that got left out.

View on HN

Recently I've noticed that the GitHub and Stack Overflow scraper clones will often be the only result for this kind of query. It looks like the blackhats have found a way to rank higher then the content they're cloning. I suspect this has the side effect of tricking Google's anti-spam system into punishing the canonical domains and URLs, because it thinks they're copies of higher ranked content.

I never peaked at the gh source but from using it I think you only see closed issues after hitting a select box or button. Maybe googlebot isn’t “doing” that, triggering the JS or whatever drives it.

This one is an open issue.

Stackoverflow does not even need scraper, themselves publish a dump.

Ah, but that's quarterly at best :)

It seems to me that the first-page prevalence of scummy clone sites can only be explained by either extreme ineptness or profit-driven malice from google. It's not like google's search engineers have never heard about this problem.

It happened to me also, not just for GitHub, but also Stack Overflow. I no longer rely on Google, and search the sites directly.

Sometimes I can't find anything non-commercial. For instance, I wanted to find out where the phrase "milk and honey" comes from. Googling it only yields a book for sale. But a Wikipedia search for it, which is one click away in Firefox, gives me this page [0] which is exactly what I wanted to know.

Looks like Google no longer acts on the mission to organize the world's information, but focuses on making money.

[0] - https://en.wikipedia.org/wiki/Milk_and_Honey

I add the keyword "etymology" to searches like this and it always gives me the answer inline. Maybe you're relying a bit too much on Google reading your mind.

This specific query was an example. I feel that everything is getting more commercial.

It seems like people with skills and ambitions don’t want to work with search. The search feature of Github is also garbage. The ranking is ridiculously bad. Stackoverflow search is also a joke. Almost forgot about the docs at MS. Clone and use a tool to find in files for MS docs will beat the MS docs sites. Also, I challenge anybody to write a serious version of find in files on Windows that is slower than the native one.

My tip is to try to avoid searching. Ask yourself where the information may be located. Then try to find that location if you do not already know it.

I also frequently hit timeouts for particular search strings no matter how much I retry.

There's not much money in it. But some people are working on search:

https://kagi.com

Having used the beta, I can say it's pretty good.

Will normies use it? Probably not. Normies will use The Google until the end of time because The Google has done a remarkable job convincing everyone that they have a unique omniscience. But if you know better, then give Kagi a chance. In my disillusionment around DuckDuckGo, Kagi has been a breath of fresh air.

I have come across this before, searching for a particular problem but not finding the relevant Github issue in google's search results, which made me search using Github's own search functionality inside the relevant repo to find what I'm looking for. I'm not sure why google wouldn't be indexing Github issues though

My guesswork based on the fact my URLs my crawler became aware of so many GitHub URLs I had to add special logic to exclude URLs that look like commit hashes:

GitHub is absurdly large in terms of the number of documents since there's ostensibly a "document" per file and commit, it's likely the crawling budget Google affords GitHub simply runs out before they've crawled it all unless they implement special logic to prioritize issues and about-pages.

> but somehow I expected that Google will find my ANY public string on the internet.

Is it the case tough? Does Google index everything? I don't think it's the case.

They will crawl almost everything (where not explicitly prohibited), however, indexing is dependant on several factors and not guaranteed.

I would personally expect them to index all public GH issues, though.

I think they used to index everything they found. But WWW has grown incredibly during the past 10-15 years.

Google doesn't index everything and they don't even return results from their index with perfect precision. You have to make some trade-offs at that scale.

I believe it's quite simple: Ad money. Those search queries that cannot reliably be monetized with ads are more or less worthless to Google. So they skip indexing those search terms. That probably also implies that search queries where the majority of users use an ad-blocker (that's us developers) will be a very low priority.

I'm not even sure I can blame them. Creating the search index is a huge investment on Google's side, both in terms of time (can't crawl too fast or else they knock the target offline) and in terms of bandwidth, CPU power, and storage space. But Google needs to operate their search profitably, on average, or else they will go bust. In the long term, I predict that this leads to ever more SEO spam and ever more ads as the internet grows and more and more NLP and AI processing is needed to separate the good results from the bad ones. Filtering gets more expensive => more false positives + need more ads to pay for it.

I've been trying hard (maybe a bit too hard?) to stir up a discussion about people actually paying for the search index, because that would allow them to get the results that they want - private and ad-free - no matter if those results are monetizable with ads or not.

In my opinion, that would make for a nice open source project. In case you're curious:

https://news.ycombinator.com/item?id=30374611

https://news.ycombinator.com/item?id=30361385

I'm not convinced. Because if google becomes only search for ads, it's almost useless.

It does find other issues tho. And google certainly has the resources and to contain all the text on the web.

Not that github is small/niche site that is irrelevant...

Moreover there are updates on that page lately, so it should be fresh content.

Anyway, maybe it's just that not enought incoming links to that issue? Maybe their algorithm has simple flaws. Maybe some search node is down or whatever. Anyway, I wanted to raise awarness on this.

Bing has this indexed, so I'm assuming DDG is finding it from there. (Bing is also, unfortunately, also returning results that do not have the string present, which depending on how one tunes one's success metrics is worse... Remember the bad old days of sifting through search hits to not find the information the engine claimed was on the page?).

There's not a lot one can glean about the operation of a search engine from one anecdote; any number of things (including the backing store on which the one copy of this data is indexed being temporarily unavailable) can explain a failure-to-find like this.

But this story does point to both an advantage a meta-search like DDG has and a best practice: different search engines will give different answers. Keep several in your back pocket; don't assume Google is omniscient.

(googler, opinions are my own. No insider knowledge about this discussion.).

Bing and Github are both owned by Microsoft. Maybe there are different scraping/throttling rules in-place on Github between Bing and Google?

Doesn't look like it.

https://github.com/robots.txt

It may also be server-side instead of client-side, which we wouldn't be able to see out here.

If Bing has a dedicated crawler for GitHub (or a back-channel to crawl its databases directly).

I suspect this boils down to those type of searches do not make money.

It is clear Google has issues with indexing Github. Couple years ago they somehow retrieved github mobile page instead of desktop page. As a result, no comments, no readmes, no code indexed at all.

A solution is to move most of the documentation to self-hosted website, leaving only references in the code itself.

If it can't be turned into a purchase by the end user Google has no reason to index it fully. It's just waste heat that doesn't generate ad revenue.

Google only indexes a fraction of the public internet.

Microsoft recently started putting github behind a reddit/twitter/Instagram login wall: comments for open source community projects are truncated unless you log in. Maybe related?

Oh wow, that sounds terrible - do you have a link that demonstrates the behavior? Was their HN discussion of it at the time? This is pretty big news that I missed.

I'm not seeing it now. I probably misinterpreted the login prompt at the end of a thread or it was an A/B test.

I just tested this on the following pages (logged out and then logged in):

https://github.com/intellij-rust/intellij-rust/issues/3757

https://github.com/webdesus/fs_extra/issues/12

Comments were not truncated.

I'm not signed in and this certainly isn't the case for me (desktop Chrome on Windows 10).

Many login walls can be bypassed if you set your user agent to GoogleBot. You may also have to use an IP from Google, which is easier now thanks to GCP.

Reply on news.ycombinator.com