The exact advice depends on how you’re running your services. My starting advice, for cloud, is this:
1. Run in multiple, separate accounts. Don’t put everything in one account. This could be as simple as having a beta account for testing and a separate production account.
2. Use cloud-based permissions systems when possible. For example, if you are running something on AWS Lambda, you create an IAM role for the Lambda to run as and manage permissions by granting them to the IAM role.
3. If that’s not possible, put your credentials in some kind of secrets management system. Create a process to rotate the secret on a schedule. I’d say 90 days is fine if you have to rotate it manually. If you can rotate it automatically, rotate it every 24 hours.
4. Set up logging systems like CloudTrail so you can see events afterwards.
Finally, as a note—people at your company should always authenticate as themselves. If you are TheBigDuck234, then you access your cloud resources using a TheBigDuck234 account, always.
This is just the start of things.
Or have a 'sudo' TheBigDuck234-to-AdminAcct mechanism if possible, or TheBigDuck234_admin account.
On my Linux machines I do sudo-to-root, but on macOS, my daily driver account does not have sudo access so I have to first su to "admin" and then can sudo from there (for GUI requests I enter "admin" (or whatever)).
There are some rough edges around the experience here if you really do want the best security posture, but you don’t have to go all the way. You can just create your one IAM user (just one per person) and then create multiple roles. When you log into the console, you authenticate as the user and then choose the account + role you want to use. I recommend creating a “read only” role. The purpose is to let people poke around in the console and debug problems without risking creating problems in production infrastructure—this is more of an operations than a security problem, though.
- It let's you decrypt same file using multiple credentials/keys (every team member has its own)
- it can use cloud vaults for encryption/decryption - for instance, keep your keys in Azure Key Vault or similar, and let the team access that using their own setup of AZ cli and SSO login you use to interact with the cloud anyway
- it will be able to keep the encrypted file semantically correct, so you still can use linter checks on push to git, etc
If you don't want to commit/share secrets you could avoid sops and put this in your direnv envrc: `[ -e ~/.local/secrets/myproj.env ] && source ~/.local/secrets/myproj.env`
I wanted a tool that allowed me to store secrets safely without tossing them in plain text env files called `sops-run`. It manages yaml manifests to store your environment variables based on the name of the binary you're running, and only applies the environment variables to the context of app you're running. I never did tidy this up into an installable python package so it can't be easily installed with pipx yet (I keep putting off finishing all of that, pull requests welcome ;-) ), but I like it better than simply using direnv or equivalents, since it doesn't load the environment variables into the shell context, though it could probably be combined with it to hot-load shell aliases for the commands you want to run.
[1] https://github.com/carlpett/terraform-provider-sops
* With the standard caveat that you should audit any software you're planning on giving access to your most valuable secrets.
For an easy, slightly hacky version I've used git-crypt (https://github.com/AGWA/git-crypt) with tiny teams. You'll need to share the decryption key (e.g. via 1password shared vaults).
As your need for security grows (but you're still not working with a giant team) you're better off using non-committed .env files locally (with need-to-be-shared dev keys stored in shared vaults in 1password) and prod GCP/AWS secrets managers remotely.
Once you work with a bigger team you'll need to start minting API keys limited in scope to each individual, for them to work with locally. The prod keys will only live in the remote environment, managed by some kind of secret manager offered by the platform and will need to be rotated frequently.
Not really. It also supports keeping the symmetric decryption key encrypted with the GPG key of each added user (and handles this automatically). This is the default behavior.
What you're saying also works (quoting from readme, emphasis mine: "Alternatively, you can export a symmetric secret key, which you must securely convey to collaborators."), but feels worse from a security point of view.
I ask because I'm currently using a scheme where I have a .secrets env file that is .gitignore'd from the repo but it has a corresponding .secrets.gpg file which isn't, but it's a pain to synchronize these; I suppose this is the problem `sops` solves...
Some of them have secrets management built-in too, like:
They also have a plugin system[3] that makes it work moderately seamlessly with various other CLI utilities which expect credentials in the environment, such as the `gh`[4] CLI and bazillions of others
1: https://developer.1password.com/docs/cli/
2: https://developer.1password.com/docs/cli/secret-reference-sy...
3: https://github.com/1Password/shell-plugins#-see-it-in-action
4: https://github.com/1Password/shell-plugins/tree/main/plugins...
Edit: to expand on this a little, even the image [0] they show on their secrets management landing page is baffling. It's showing an entry in the 1Password app called `AWS - Access Key`, which for some reason has a username and password. Now if I need to inject that into the environment variables on my server, what's the name of the envvar, because `AWS - Access Key` isn't going to work. How do I separate staging variables from production variables? How do I know which project this is the AWS access key for?
[0] https://images.ctfassets.net/b71sid4v0oel/7zRNbDUY8dxGuKxUtV...
1p lets you create arbitrary fields in an entry, so you could create one with a VARNAME field or something, or just use a naming convention on the entry. for project isolation you could create separate vaults or use tags to distinguish.
So on a dev machine it could use the dev vault for your project, but when you deploy it could use the prod vault.
Within vaults you can name and organise how you like, you’re not limited to usernames and passwords. You can have arbitrarily named fields, whole text blocks, or files.
Inspiration here: https://gist.github.com/bmhatfield/f613c10e360b4f27033761bbe...
Then you can share a dotfile that includes:
export OPENAI_API_KEY=$(keychain-environment-variable OPENAI_API_KEY)
And share/set the variable in Keychain.
This way, the opportunity to expose the secrets is more limited to the actual run-time of the application. You don't need to risk exposing your secrets every time you git push.
For example, my .env file may have something like this in it:
DB_PASSWORD = @AWS::db_password
Whenever my library reads a value that begins with `@AWS::`, it knows to resolve (and cache) that value by querying AWS's Secrets Manager at runtime and looking for the config setting set there (`db_password` in this case).
This is nice because I can check-in these .env files since they don't contain anything sensitive, but still gives me the flexibility to hard-code in secrets when working locally in my dev env.
Probably annoying if you have more than one machine though
I ask because my research suggests that there's a class of security vulnerabilities where attackers can read arbitrary files - but since /etc/system/systemd can be limited to be only readable by root, and the services it runs started by other less privileged users, I wonder how bad it would be to store a plaintext secret right in the .service file would be in practice. Especially since it seems this credentials management thing seems to just create a directory for the process with the decrypted passwords readable anyway (although maybe that's still not readable by an attacker? Still trying to figure this all out myself).
Among other things, TPM and TPM2 are physical chips, which means even someone who steals your actual hard drive couldn't actually decrypt your stuff unless they also somehow got access to the rest of the computer containing that TPM chip. Huge improvement, although I'm not sure if your run of the mill cloud VM has (or even could have) such a chip permanently and uniquely bound to them.
There is a script in the repository that will bootstrap your local env by storing encrypted secrets into ~/.config/ by asking some questions that you get from the shared passwords manager for development credentials. The key to decrypt them is password protected and requested at application boot.
1. Stop using API keys. Configure SSO integration for developers and OIDC for automation. For example, this is very easy to setup with AWS.
2. If the above is not possible, then store credentials encrypted at rest. Decrypt them only at runtime. For example, SOPS to store encrypted credentials into the repo, then AWS KMS holds the decryption key. The SOPS Readme is very helpful.
The best work-around I could come up with (not having a Hetzner account to actually kick the tires upon) is that you could inject a private key that you control into the instances via cloud-init (or volume attachment) and then sign any subsequent JWT using it. For sure it would not meet all threat models, but wouldn't be nothing either. I was hoping there was some chain of custody through Vault[2] but until Hetzner implements ANY IAM primitives, I'm guessing it's going to be a non-starter since the instances themselves do not have any identity
1: https://docs.hetzner.cloud/#server-metadata
2: https://github.com/hashicorp/vault/blob/v1.14.7/website/cont...
It's a bit different than most of the other tools listed here, in that it is designed to generally solve the papercuts of dealing with config (both sensitive and not), and is not coupled to storing your sensitive config in a specific platform (paid or otherwise).
You define a simple schema for all of your config, and you get validations, built-in documentation, full type safety, and the ability to compose config together in any way you choose. You can also easily share config across a monorepo (if you are using one).
Additionally, our drop-in integrations (node, vite, nextjs, astro, remix, more on the way) go a bit deeper and do things like help you detect and stop leaked secrets, redact secrets in logs, and deal with the footguns of boot vs build time config in hybrid rendering environments.
As for storing/syncing sensitive data, we currently have 2 plugins but more are in the works and will be guided by user demand. The first lets you store your secrets encrypted within your repo (like dotenvx, git-crypt, etc), and the second lets you sync with 1password. Personally we think the 1password plugin makes sense for a lot of teams, since they are probably already using it. You can wire up individual items to your schema, or pull from dotenv style text blobs. You can (and should!) segment items into multiple vaults, and use multiple service accounts to access them. You can even mix and match plugins to pull secrets from multiple services.
(see https://dmno.dev/docs/plugins/encrypted-vault/ + https://dmno.dev/docs/plugins/1password/)
In the future, we'll have deeper support for things like key rotation and single-use keys, k8s, way more backends, etc. It's all open source, so come and tell us what you need (or even help us build it) and we'll make it happen!
If it's not obvious already, I am the creator :)
PS - Feel free to hit me up for more info or a demo - theo at dmno dot dev
I advise against dynamic secrets. In my opinion deployment should be immutable. If you need to change secrets, you need another deployment. The usual exception is where the deployment is too costly or you can't do zero-downtime blue-green/canary deployments.
The template file can be something like a .env.j2 and the secrets can be pulled from something like hashicorp vault, which enables you granular permission for the pipeline runners to read just the necessary kinds of secret that particular deployment needs.
You need however to put a little effort into creating these pipelines, but the benefits are huge.
The problem with `.env` files is that you're leaving credentials unencrypted on disk and it's easy to leak these files during screen sharing and with multiple projects there will eventually there will be so many secrets spread/sprawled everywhere that you lose track of what credentials are being used and what are expired. You want to be able to inject the required keys only when needed and leave no trace behind when not needed.
I wrote about this in our documentation for Polykey: https://polykey.com/docs/how-to-guides/developers/developmen...
We are still working out the kinks but I expect that one should be able to easily do `. <(polykey secrets env project-vault)` in whatever development shell you have, perhaps even reference a schema for expected keys.
I see references to gitlab CI yamls in the source tree, implying you are multi-CI which seems like an extra awesome way to showcase the one source of truth for credentials. Bonus confetti if Ploykey were able to auth as the CI job via https://docs.github.com/en/actions/security-for-github-actio... or https://docs.gitlab.com/ee/ci/yaml/#id_tokens
Using Polykey in the CI/CD situation as you point out in your links is actually one of the major complex usecases we designed Polykey for, however it's actually quite a complex problem domain. We expect to do a sort of "painkiller" phase 1 first where PK is used as the network for sharing secrets, and then a subsequent "vitamin" phase 2 where secrets are no longer shared at all, because authority is delegated through trust federation.
Right now PK is a decentralized secret sharing system (every agent is a P2P node), so there's no well-known trust anchor to form a trust federation via OIDC. However once we have PKE ready, then we plan to enable OIDC customer portals within PKE following ideas from https://openid.net/specs/openid-federation-1_0.html (e.g. yourcompany.enterprise.polykey.com). This requires a more sophisticated policy-logic system integrated into each Polykey node's sigchain, atm we have work in progress for public/private network segregation.
In our internal documentation, I have a diagram of how PK would integrate into a CI architecture, just haven’t shared it publicly yet. I’ll try to get it out soon. Let me know if you’re interested in more details!
to prevent this, use:
$ dotenvx ext precommit --install
https://aws.amazon.com/secrets-manager/ https://keepersecurity.com/
What is a long-lived user account? https://g.co/gemini/share/84c224b18bf0
Check out https://onboardbase.com/. It's pretty handy for keeping secrets secret.
https://github.com/Swoorup/passage-nix-secrets-template
EDIT: This is meant to used in a nix-based deployment setting, and also you don't want to commit the identities file unless you use yubikeys (Something which I forgot to mention in the readme).
1. Load secrets dynamically at runtime
2. Share internal creds via e.g. 1Pass
Environment variables (+managing them with .env files) are a better start than putting keys in your codebase, but this can also be leaky/hard to keep up to date.
Most cloud providers have some sort of secret management tool. Vault by Hashicorp is another solid option if you want to run your own.
If you’re hosted on AWS, I’m personally a big fan of Credstash[0], which is basically a simple wrapper around DynamoDB+KMS.
Cheaper than the AWS Secrets product and fast enough.
I previously built a config that would take secrets from Credstash, env vars, and .env files (in that order). This offered the best of both worlds for local and remote deployments.
If you want something simpler, have a look at SOPS: https://github.com/getsops/sops
You still have to "manage" the vaults' passwords, though.
For users use federated auth/sso with temporary credentials or static user keys that require 2fa with aws-vault.
git-crypt - https://www.agwa.name/projects/git-crypt/
Or
blackbox - https://github.com/StackExchange/blackbox
If you don’t, .env files with a proper .gitignore
Or AWS secrets manager, GitHub secrets, Azure KeyVault and inject them.
If you run on something like Azure, use Managed Identities. Passwords shouldn’t be used period. API keys should be securely injected in a pipeline from a vault.
At work we use Ansible to fetch secrets from Hashicorp Vault.
This way credentials leak will not do much