Quickly find sensitive files in your GitHub repo | Hacker News Reader

Quickly find sensitive files in your GitHub repo

about.sourcegraph.com

34 pointsjdorfman17 comments

One thing to be aware of: Tools like this (and indeed, this one specifically) tend to take what I can only describe as "what AWS keys look like _now_" mentality to how they construct regexps.

I learned this the hard way due to a script bouncing around an internal gitlab repo for ages with hardcoded AWS keys, and the second someone accidently published it to a public gist, we learned very quickly people that will abuse public AWS keys actually use AWS's recommended regexp ( https://aws.amazon.com/blogs/security/a-safer-way-to-distrib... ) to find AWS keys, even old ones, though it does result in more noise, which is no doubt why tools like this try to be more specific.

One technique API providers could adopt is prefix all keys with a description so there's no ambiguity between keys and random base64 strings and it would make automated checks a lot simpler: e.g. amazonsecret-... githubsecret-... donotcommit-... etc.

You're definitely on to something.

I know (personal) Github API tokens use a prefix of ghp_ and the Github-app JWT-issued tokens have another prefix (maybe gha_? Can't remember)

As mentioned recently here on HN, a generalized version of this technique is what RFC8959[1] proposes via "secret-token" prefix.

Do add these to all your pre-commit hooks and other password detectors/log analysers!

[1]: https://www.rfc-editor.org/rfc/rfc8959.html

This is precisely why github switched their token format and what they recommend other service providers do:

https://github.blog/2021-04-05-behind-githubs-new-authentica...

I agree, some do but more should dare I say need to follow suit.

This is great thanks for putting it on my radar.

No relevant results for "master.key" (first and only thing I tried searching for), so I assume they have figured out some awesome way to detect these files based on their contents. Or the heuristics are closed-source.

https://github.com/search?q=org%3Asourcegraph+master.key&typ...

https://medium.com/@thorntonbrenden/rails-and-the-legendary-...

Are there any tools that check whether you're pushing keys and secrets BEFORE you've uploaded them to Github and it's already too late? E.g. checks your commits, and gives you a massive warning when you've committed something you're not supposed to? Even better if it prevents your commit from going through unless you force it

How about a pre-commit hook?

https://github.com/pre-commit/pre-commit-hooks#detect-aws-cr...

Although a pre-commit hook exists for detecting AWS credentials, I can imagine other checks (e.g ssh credentials, passwords, etc)

You can use commit hooks to script git

For people in the JS ecosystem theres this eslint rule: https://github.com/nickdeis/eslint-plugin-no-secrets

Github itself has a product like this. It's better than the pre-commit hook approach since developers can turn that off (or never install it in the first place if you don't roll it out via MDM), and this centralized approach can't be bypassed.

But, even if you're on their enterprise plan it's still an additional addon (and expensive):

https://docs.github.com/en/enterprise-cloud@latest/code-secu...

There are a ton of them, here are a few:

https://martinfowler.com/articles/session-secret.html https://github.com/trufflesecurity/trufflehog https://github.com/awslabs/git-secrets https://github.com/sirwart/ripsecrets https://thoughtworks.github.io/talisman/ https://github.com/michenriksen/gitrob

I've never accidentally committed a secret into any repository. When I create a file that holds secrets, first thing it either gets encrypted or goes into gitignore.

What exactly is the path that leads to committing secrets, apart from utter incompetence or disregard? Do people not review what they're committing at all?

Every time I've done this, I've done so knowingly and intentionally, into a repository that is not public and will never be public.

That last part is what makes it okay. I can only assume the kind of person who puts their dotfiles in a public github repository - or, more generally, puts things on github by default - does not have as strong a public/private distinction as I do.

Yes of course, I started my second paragraph wrong - it should have included "or no plans to ever make the repo public"!

You've never accidentally committed secrets into a public repository, right?

Not that I know of!

Reply on news.ycombinator.com