"Production keys in source control" is right up there with "mistaken routing table entry" and "fat-fingered DNS config" on the list of critical company-breaking mistakes that you'd think would be easy to avoid, but aren't.
I have git hooks to lint code syntax, but nothing for scanning for leaked credentials. Looking @ TruffleHog now, mentioned by another poster.
I mean, I'll bet Toyota knew this organizationally. They had security people sign off on the design who all knew how secure key management is supposed to work. They probably reviewed this github release. And it happened anyway.
Maybe they weren't supposed to be production keys. Maybe it was a development key from someone's early cut of the tooling that got mistakenly reused for production. Maybe a script that updated all the keys mixed up which was which.
The point is that the existence of a Clear And Unambiguous Right Thing to Do is, in practice, not sufficient to ensure that that thing is done. The space of ways to mess up even obvious rules is too big.
And that's surprising, which is why (1) it keeps happening and (2) people like you don't take the possibility seriously in your own work.
At my org, we even try to generate all secrets with a standardize prefix/suffix so as to make them very greppable. That doesn't stop "Architects", "Customer Solutions", "Analytics" types from … just working around the standard tooling and manually generating one by hand because … IDK, they think they know better? I really don't get it.
IIRC they even basically got some hospital admin fired for creating a hand washing mandate, despite it being proven to save lives.
https://www.npr.org/sections/health-shots/2015/01/12/3756639...
(talking centuries ago, but maybe even today)
Looks like its still a "recent" issue, lol https://www.nytimes.com/2006/09/24/magazine/24wwln_freak.htm...
Now, it's an embedded maps api key, there's no cost to use it, nobody can use it from a domain other than mine, and it's easily visible in the page source if someone views that, so there's really no reason not to commit it since even if I didn't commit to somewhere publically, it's still publicly available on my website and nobody else can use it anyway.
Is this a reasonable exception to this rule?
You could use Google actions to build your pages site injecting the api key at build time. It’s stored a repo secret rather than in code. Of course since you deploy the site publicly, the key will still be visible.
1. someone adds new service/server/infra in a submarine manner
2. it goes to prod
3. the cert expires and outage begins
4. my team is asked what to do, because "we're the cert experts"
5. we add it to the monitoring
So it only happens once … per service. Which isn't great. But how do you get people to slow down and do simple shit, like add a monitor to your new service? (Or promulgate a design doc…)
This still doesn't protect against the vector where somebody else in the company has managed to prove themselves to be responsible parties to another CA/issuer.
And, like I said, usually it's someone who doesn't grok certs doing it without asking for help in the first place, so they're not going to get why ACME. (Because I am tired of doing cert renewals. I've had enough for a lifetime…)
Your processes are failing your development teams, and you need to fix them, rather than blaming your teams, which achieves nothing.
For example it's one line for me to configure a new machine infrastructure built to have a certificate for myservice.myorg.example, and there's a Wiki page reminding me what the line is, or I can look at lots of services which already work. If I do that, the automation happens from the outset, my service never has a custom cert or lacks monitoring, it has monitoring from day zero and its certificates are automated. I happen to really care about ACME and the Web PKI and so on - and would have gone the extra mile to do this, but I was astonished on Week One at my current employer to realise oh, this is just how everything works here, the Right Thing™ is just easier.
Does your company have a Wiki page saying how to do it wrong? After writing the page about the right way, update the bad wiki page with a link to your new page, and cross through all the previous text, or even just delete it.
If you have firewall rules or a proxy, block random unauthorised stuff. This is probably a reasonable strategy anyway. Now they come to you to unblock their "submarine" service, and before you do that's the opportunity to insist on proper certificate behaviour.
But then we get some vendored product that manages to be completely unable to run in k8s, devs avoid the automation for $reasons, etc.
> Does your company have a Wiki page saying how to do it wrong?
Sometimes we do! I've found a few of these after really pressing the point of "why are you doing it this way?" hard enough. But you have to a.) get an answer to that and b.) the answer has to reveal they followed some shadow docs.
> If you have firewall rules or a proxy, block random unauthorised stuff.
Your rouge service implementer just creates their own VPC; they are in control of the firewall.
Should my security team either set appropriate privileges or delegate that to my team? Perhaps. I have to get them to adopt like RBAC and ABAC first; they fervently believe our industry's regulations forbid "job function begets access" (i.e., RBAC) type policies. (They desire that, even if a job function begets access, that if you're not needing to exercise that access, it should be revoked until such a time that it is required to be exercised. But this means that you end up with "all security reqs. must flow through security team" style thing, and there are then a lot of them (because they are so ephemeral) and so any process must inherently be ignorant of whether the request is right. So your rouge implementor's request for "I need to implement $high level service" is basically carte blanche.
The thing about shadow-docs and shadow-services is that they're hard to find out about in a timely manner. A lot of these comments are fighting the very core of human nature.
(We used to be better about this as a company, back when we were very engineer heavy, and filled with good engineers — most better than me. The quality bar definitely fell at some point, and we've hired a lot of not engineers doing things that really would be better served by an engineer. Y'all are working at rainbow companies, and I don't know how to keep a company in that state or move it to that state as a bottom-of-rungs eng.)
2. Email everyone who might be involved a note about this and a link to the documentation and why it is important.
3. Next time someone ignores it, rip them and their manager a new orifice.
4. Wait for word of #3 to spread.
Might help...
We do (now) follow the CT logs for ourselves. That catches some cases, but not everything.
The wildcards would be tough but you could follow cnames as those would need to be in the cert as is.
IaC, right? If you don't put keys into the Code you can't have Infrastructure as Code. Without keys the code only partially defines your infrastructure.
Assuming you have been given the right tooling, there is no reason for it to be in source code.
Because EnvKey integrates tightly with environment variables, no app code changes are needed to switch, so it only takes a minute or two to import/integrate a typical app.
EnvKey is designed to help avoid incidents exactly like the one that just hit Toyota, while staying out of your way and even making your life significantly easier as a developer by simplifying many aspects of config management.
Give it a look if you know you have some room for improvement in this area and are looking for an easy, secure, open source, end-to-end encrypted solution :)
However, I'm currently working with a group using Terraform on GCP (GKE), and it's popular with them to use Secret Manager to manually create a secret in there (when it cannot be auto-gen'd with the IaC, a fairly small subset of things) and then reference that secret from the infra-defining code.
I think of it as being akin to "this service requires a correctly configured FOO_BLAH variable in it's environment". I don't really see it as any failure of achieving some IaC goal, but defining infrastructure code isn't my primary function, so take this with a grain of salt.