Grep.app – Search across a half million Git repos | Hacker News Reader

Grep.app – Search across a half million Git repos

grep.app

189 pointsthunderbong30 comments

Please don't check your keys or passwords into github.

This was my first search to show AWS API Keys. [1] and my second to show google API keys. [2] Unfortunately, there are many more of these types of searches - and far too many results.

[1] https://grep.app/search?q=AKIA%5B0-9A-Z%5D%7B16%7D&regexp=tr...

[2] https://grep.app/search?q=AIza%5B0-9A-Za-z%5C%5C-_%5D%7B35%7...

As far as I can tell, all the git repos come from GitHub, and GitHub automatically sends you a warning if you commit access keys of various kinds (and I seem to remember they even de-activate some, maybe AWS keys, automatically? Not 100% sure about this). Most of the keys shown are unlikely to be working, especially the ones older than just a couple of days (either owner notice, or someone else was faster than you, got access and regenerated new keys for themselves)

> or someone else was faster than you, got access and regenerated new keys for themselves

There are many people who do these types of searches - regularly. If there is any key material checked into gh, consider it immediately compromised, even if the repo is removed.

Tons of automated scraping for these sort of things as well, with the whole flow of "check if key is working, check permissions, if authorized, generate new key and deactivate old one".

Yes. There are indeed several mitigations in place. However, my point was that within 5 seconds of looking at this tool I was able to uncover hundreds of potential issues. There are many other regexs I have on my list. Please don't fall into the trap of "we have anti-virus, so we can't get attacked."

Sorry if I was unclear, my previous comment was about attackers doing that in order to compromise infrastructure, not that companies are doing it in order to defend themselves. But companies probably do it to defend themselves as well, but I'm not as familiar with that as with the other side of things.

GitHub PM here, just confirming that that's correct (re: us sending detected secrets to the relevant service provider, who take action automatically). There's more detail here: https://docs.github.com/en/code-security/secret-scanning/abo....

This is probably one of the single most important security measures on the whole internet.

Thanks for keeping an eye out.

Most of the Google API keys look like they are client side keys and should be configured to only work on specific origins or apps.

Your search has a lot of false positives like AKIAIOSFODNN7EXAMPLE

Most of these actually look like fake keys to mock data in unit tests.

Hacker noises - "I'm in"

----- BEGIN RSA NOT SO PRIVATE KEY -----

Many of these are tests, false positives (e.g. base64 images or "Migration1599570560FixSlovakiaDisplayedAsSloveniaTest"), or examples from documentation.

Can't find a single match for an actual API key.

These are probably invalid by now.

https://grep.app/search?current=2&q=api_key%20%3D%20%22%5Ba-...

As others have posted, Github implemented an alerting system for keys/passwords/etc. that have been checked into the codebase. My guess is most of these will be invalidated by now, either by the developer requesting a new key (and not publishing it), or the provider invalidating it for them.

https://sourcegraph.com/search?q=context:global+AKIA%5B0-9A-...

Sourcegraph is much better than this site.

I'd be curious to hear how the ingestion/search pipeline works — the searches are incredibly fast!

It uses Solr: https://news.ycombinator.com/item?id=22397392

Previous discussion: "Show HN: Search code in GitHub repos using regular expressions"

https://news.ycombinator.com/item?id=22396824 (614 points | Feb 23, 2020 | 155 comments)

Recent and probably related:

The technology behind GitHub’s new code search - https://news.ycombinator.com/item?id=34680903 - Feb 2023 (168 comments)

This is a great idea and I commend it and I hope others find it useful. However, I have a rather narrow use-case for GitHub searches, and in my initial experiments, this tool comes up 100% empty on common searches I use every day for work.

Most of my GitHub searches don't actually revolve around actual code-code. I search for text strings in prose that turn up in code searches. So the file types may be text or MarkDown instead.

Perhaps grep.app only indexes and presents actual code-code and filters out all the prose I'd be interested in. Sad, because there are actually a few instances where my searches involve code with many intricate special characters, and GitHub's current code search ignores all those.

Previous discussions:

- https://news.ycombinator.com/item?id=28424845 (September 5, 2021 — 54 points, 6 comments)

- https://news.ycombinator.com/item?id=22396824 (February 23, 2020 — 614 points, 155 comments)

Missed opportunity to call it "Grit."

At first glance, this is great!

Haha I can share my first use case - searching for repos I'm involved in, kind of like searching for myself on Google Scholar

Search across a half million GitHub repos. It doesn't support any other git hosts than the proprietary one.

Yet it doesn't have a clue about the existance of one of my 17 star GitHub public repos

How Is this different than GitHubs search?

Hi, I work on GitHub code search. Grep.app is based on Solr and indexes about 500,000 public repositories. As far as I know, the author has not shared their tokenization strategy, but based on the the results I expect they are indexing trigrams to perform regular expression matching.

The current GitHub code search is based on Elasticsearch and indexes more than 100M repositories. Its tokenization is based on whitespace, case changes (like CamelCase) and punctuation (like kebab-case) and strips out non-letter characters like < or { which is why it can't do exact match search.

Our new code search, currently in beta and indexing about 45 million repositories, uses an search engine we built in house that indexes content using a technique we call sparse ngrams. This allows us to execute searches faster than a trigrams index, while also being smaller than a positional trigram index. My teammate discussed some of the technology behind it in our blog post that was published yesterday: https://github.blog/2023-02-06-the-technology-behind-githubs...

GitHub can be searched directly without any JS bloat.

Reply on news.ycombinator.com