I felt like a bit of a dope but it was neat to have it happen to me. Lesson learned for sure.
https://docs.github.com/en/code-security/secret-scanning/sec...
Next time I implement api keys I wonder if it’s worth going out of my way to make them easy to identify. Eg, by prefixing every key with a few well known characters. Like FMLA_xxxxx for a fastmail app key.
https://github.blog/2021-04-05-behind-githubs-new-authentica...
We will be applying thanks for sharing greystell
Yeah absolutely - I'd love to hear about it!
https://docs.github.com/en/developers/overview/secret-scanni...
Your dev team would probably be able to give that a glance and estimate the work.
It can be as simple as a script you have to run once, but it can't be automatic. Which also means you can't really trust contributors to do it, even if they're well-meaning some will forget.
https://docs.github.com/en/enterprise-cloud@latest/code-secu...
Also the “you” below is the generic you - not you personally.
Disclaimer: I work at AWS in Professional Services, all rants are my own.
Now with that out of the way, I hate the fact that there are way too many code samples floating around on the internet that have you explicitly put your access key and secret key in the initialization code for the AWS SDK.
s3 = boto3.resource(‘s3’,aws_accesskey_id=ccxx,aws_secret_access_key_id=cccc)
When all they have to do is
s3=boto3.resource(‘s3’)
All of the SDKs will automatically find your credentials locally in your .config file that is in your home directory when you run “aws configure”.
But really, you shouldn’t do that, you should use temporary access keys.
When you do get ready to run on AWS, the SDK will automatically get the credentials from the attached role.
Even when I’m integrating AWS with Azure DevOps, Microsoft provides a separate secure store that you can attach to your pipeline for your AWS credentials.
But yeah, if the wrong thing is easier or more straightforward than the right way, people tend to follow it when they have a deadline to meet. To end on a positive note, at least cli v2 makes bootstrapping the credentials to a workstation a tad easier!
I know I should know this seeing that I work in ProServe at AWS, but what do you mean?
I’m going to say there is never a use case for embedding credentials just so I can invoke Cunningham’s Law on purpose.
But when I need to test something in Docker locally I do
docker run -e AWS_ACCESS_KEY_ID=<your_access_key> -e AWS_SECRET_ACCESS_KEY=<your_secret_key> -e AWS_DEFAULT_REGION=<aws_region> <docker_image_name>
Sometimes you might need to use stuff that for some reason fails to use the envars, I think I’ve bumped into some stuff which reads s3 via self-rolled http calls. Dunno if it was to save from having boto as a dependency, but those things are usually straightforwardly engineered so no logic in figuring out the other, more smart ways to handle the keys. Here are the parameter slots, enter keys to continue.
I wasn't familiar with how AWS credentials are usually managed so I was very confused why I had to make my own struct and implement the `CredentialSource` trait on it. It felt like I was missing something... because I was. You're not supposed to enter the credentials directly, you're supposed to use the built-in EnvCredentialSource or whatever.
I’ll put things like server secrets and whatnot, there.
Of course, I need to make sure the local directory is backed up, on this end, since it is not stored in git.
Works a treat.
I am serious. If there is a better way, I'd use it.
Remember that I don't do online/server-based stuff. Most of my projects are for full compilation/linking, and rendering into host-executable, binary apps. There's a bunch of stuff in my development process that never needs to see a server.
[UPDATE] I ended up doing something even simpler. I have issues with running scripts during the build process, unless really necessary (I have done it, and will, again).
Since this is Xcode, I simply needed to store the file in a directory (still with the global ignored name) far out of my dev tree, and dragged the file into the IDE.
See, I thought that was a big strength of a lot of the AWS documentation over Google Cloud.
An AWS example for, say, S3 would show you where to insert the secrets, and it would work.
The Google Cloud Storage examples, though? It didn't seem to have occurred to them that someone reading "how to create bucket example" might not have their credentials set up.
And when the example didn't work - well, it was like the auth documentation was written by a completely different team, and they'd never considered a developer might simply want to access their own account. Instead the documentation was a mess of complicated-ass use cases like your users granting your application access to their google account; sign-in-with-google for your mobile app; and so on.
Google's documentation is better than it once was - but I've always wondered how much of the dominance of AWS arose from the fact their example code actually worked.
Just to clarify, I’ve never seen a code sample published by AWS that has you explicitly specifying your credentials. (Now I await 15 replies showing me samples hosted on Amazon)
But I can't understate how important it was that the AWS getting started guides said "Go to this URL, copy these values into this file" while Google's examples and getting started guides... didn't.
[1] https://web.archive.org/web/20120521060506/http://aws.amazon... [2] https://github.com/amazonwebservices/aws-sdk-for-php/blob/ma...
But here is the newest documentation for PHP
https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/s...
Moral of the story: make your keys regexable without likelihood of false positives!
We should have our systems and precommit hooks then alert us when those enter places they shouldn't and help us automate rotation.
And what if your DEV is not actually well isolated from PROD/other infra? And what if some real data sneaked into DEV? Etc.
Previous HN discussion: https://news.ycombinator.com/item?id=25978185
Bonus: adding SRVCE_PRVT_{KEY} and SRVCE_PUB_{KEY}.
I also think that it should look just a bit cryptic to make a person unsure if they can meddle with the string.
How long does it take to get the response vs external bots pulling the data? What mechanisms does GitHub have in place to stop bots who monitor repo changes? I ask, as I have been there and it is super scary how fast someone/bot pulls repo data changes, as in minutes, and the repo we had back then was not popular.
https://github.com/search?o=desc&q=secret&s=updated&type=Rep...
But the commit still exists locally (since git is decentralized) so you now end up with a weird state that you have code you cannot push to origin. Definitely not a desirable feature.
> at least git itself can via hooks
I already said that:
> they can block it themselves with tools like gitleaks invoked on a pre-commit hook
The problem with git hooks is that they're not cloned with the repo. So you're reliant on the user installing those git hooks locally (sure, some repos will have helper scripts to install the hooks for you. But you're still reliant on the user running that script).
If there is data that should never be pushed to origin, then it is a highly desirable feature that the server block pushes that include that private data.
> The problem with git hooks
I was talking about GitHub's own git hooks that run on their servers, not about any local ones.
> is that they're not cloned with the repo.
It would be a terrible security issue if they were automatically enabled after cloning.
It’s already too late by that point because your secrets have already left the building. You’re not relying on upstream being honourable
> I was talking about GitHub's own git hooks that run on their servers, not about any local ones.
There’s no such thing. You can have CI tooling like GitHub Actions, but they’re a different beast to git hooks
> It would be a terrible security issue if they were automatically enabled after cloning.
It doesn’t have to be either/or. There are ways of having a sensible compromise. Like a git config that enables hooks from known safe origins. Or having the user promoted whether they want to install git hooks upon cloning.
True, but it is better than the secrets becoming entirely public, automated bots could be harvesting them and exploiting the resources they protect.
> There’s no such thing.
I would be surprised to here that GitHub doesn't actually run git on their servers. If they receive git pushes using git, then own git hooks are involved, ones that GitHub has written for their own purposes. They could simply add one to block bad pushes.
> Like a git config that enables hooks from known safe origins.
That sounds a bit terrifying to me, but I'm not of the GitHub generation.
> Or having the user promoted whether they want to install git hooks upon cloning.
That sounds like it would enable phishing-like attacks and people just clicking "yeah sure" without verifying the safety of the hook.
True. And some popular repos do already run into this problem. So it’s not a theoretical problem.
> I would be surprised to here that GitHub doesn't actually run git on their servers.
They’ve documented about how their backend works so there’s no need to speculate. They run an implementation of git but not the standard git CLI.
> If they receive git pushes using git, then own git hooks are involved, ones that GitHub has written for their own purposes. They could simply add one to block bad pushes.
They have their automation, GitHub Actions.
Sure they “could” also implement what you’ve described but it’s not how it currently works. So a pointless argument since we could be here all year discussing the literal infinity of different things Github “could” do in theory but that their infrastructure doesn’t currently support.
> That sounds a bit terrifying to me, but I'm not of the GitHub generation.
What I posted has literally nothing to do with GitHub. In fact if your origin is private git server (as I started out using git, since GitHub didn’t exist back then) then it’s even easier to designate a trusted origin. This approach makes total sense for businesses. Doesn’t work so well for open source but it’s just one option of many.
> That sounds like it would enable phishing-like attacks and people just clicking "yeah sure" without verifying the safety of the hook.
Potentially yes. But if you’re cloning a git repo, making code changes and then committing it back, you’d hope that individual is competent enough to audit the git hook. At the very least, they’ll be running the build scripts locally to unit test their changes, so it’s not like that phishing attack isn’t already present. Feels very much like you’re looking for reasons to dismiss any suggestions here rather than have an intelligent discussion.