Hacker News Reader

Comment by sergioisidoro | Hacker News Reader

Twelve factor app: use environment for config

Docker: Don't use environment for config, it's insecure

I like and use many of the patterns of 12 factor, but indeed some of these were written in the context of VPS, where environment is stable, secure and more constant (unlike containers, where environment might end up shipped in one of the layers). In the age of containers, this particular point has been a bit of a struggle for me. Docker secrets don't always play well, and you need to do some gymnastics to make it work.

View on HN

Could you elaborate? How are environment variables insecure? If the secret will be present in the application throughout it's lifetime, then what's a better alternative?

Just because a secret is injected into the application as an environment variable does not mean it's handled insecurely elsewhere. For example, AWS ECS containers have built-in support for fetching secrets from Secret Manager at startup and passing them as environment variables: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...

This fetches the secret at the time the container starts up (IIRC), and uses the IAM credentials of the running application to fetch them from Secret Manager (so it must have permission to the secret).

The merit of using environment variables seems to depend entirely on how they are, well, configured. With a mechanism like this I don't see much disadvantage. (The main disadvantage I see is that generic malware that attempts to dump environment variables might capture them, but defending against that threat is largely obfuscation, unless you aren't storing the secret at all permanently in memory.)

- any sub process has full unlimited access to env variables

- env variables are prone to be dumped in various contexts and might leak secrets to e.g. logs

- env variables might leak, e.g. in context of docker build stages etc. (but then using building a docker image to build both the image and the software is often anyway not a good idea)

- env variables are often easily accessible by other processes

the method you mentioned of ad-hoc injecting a secret as env by some form of security manager for containers avoid many such problems

> - env variables are often easily accessible by other processes

A process would need to have the same permission (same user) or root to read /proc/$PID/environ no? or what mechanism is there for a process to read another process env vars (not children processes, that was already mentioned)? unless of course, a process dumped them in some world-readable file (like logs as mentioned) or some other silly way

Exactly. The operating system already offers security for environment variables: use it. And you get the benefit of increased security in other places by not sharing users between applications.

except no it does not, env variables wherent designed to be secure and the degree of secruity the system provides is like a bad joke if you take security serious

still works if your system is a single application container

otherwise it does not

I'm not sure I agree. If that were true, the security of the user system itself would also be a bad joke and it's not. Regardless, what I meant is that the OP made it sound like they're using the same server and user and container for multiple applications, but if they did things correctly, the risk of sharing environment variables across applications wouldn't exist.

The problem is the env stays readily accessible to most programs with similar permissions and all child programs if no special actions are taken.

This is (part of) the reason programs accept secrets like passwords normally only (or strongly recommended) over stdin or similar instead of env variable or cmdline.

Or why it's not rare (or was in the past, probably is today) to have a encrypted cert file and inject the password to it through a side channel.

It's also not rare for programs to dump environment variables in various situations, including to logs. Or e.g. an intrusion detection program might snapshot all programs running + their cmd arguments + their (creation) env and then run analysis on it.

And while you can use e.g. selinux and similar to add a ton of security and prevent such issues, or use systemd to run the program under ad-hoc users with minimal permissions and various isolation it's very often not done.

EDIT:

To be clear I'm not saying you can't use secrets in environment variables safely.

I'm saying by default by their design environment variables are not "secure for secret passing". But many things for which stuff like that is true can, under the right circumstances, still be used securely.

> env variables might leak, e.g. in context of docker build stages etc. (but then using building a docker image to build both the image and the software is often anyway not a good idea)

Since the rule describes runtime configuration, it shouldn't be involved with the build process at all

My comment is specific to docker, although environment as a broader sense can be discussed.

I remember reading that it is possible to end up in situations where your environments can end up in the layers of the docker image. Also, you can read the secrets of a container with inspect, and apparently linked containers might be able to access other containers's secrets.

I think it was this article: https://snyk.io/blog/keeping-docker-secrets-secure/

Presumably because /proc/$PID/environ exists and can be read by the user that owns the process — which means that any attacker that can figure out how to spawn an arbitrary subprocess running as the daemon user (a pretty common vulnerability class!) can exfiltrate all your env-vars.

It's similar to the reason that you're not supposed to supply passwords on the command-line (/proc/$PID/cmdline exists), but instead either read them from stdin, or do the big dance of

1. creating a file that contains the password, is owned by root, and is 0400;

2. having your daemon start off running as root for just long enough to read the password into process memory;

3. then having your daemon drop privileges before doing any work, so that someone exploiting the daemon won't have the privilege as the "do the work" user to read the file with the password in it.

On the other hand, without platform support there are not a lot of easy alternatives. It at least encourages separation of config and code, keeping secrets out of vcs.

Agree. The main benefit of environment variables is portability. They work pretty much the same way in every platform I have encountered. As soon as a system gets into files, it's possible to get bogged down in the complexity of Windows vs. Unix, file permissions, SELinux, etc. The simplicity is often worthwhile over the extra security theoretically possible with well-managed files.

Yeah, I've never been a fan of secrets in env vars for this reason. It is shocking how frequent dumb debug consoles have accidentally exposed env vars. You really don't want that to accidentally expose your private keys.

There's really nothing wrong with secrets mounted on the file system via k8s. But, of course, all of this depends on your deployment environment.

Sometimes env vars are the least bad option, because secrets really are always hard.

> dumb debug consoles have accidentally exposed env vars

Huh? If the app can read the secret, then the app can... read the secret. Maybe I'm misunderstanding what you call a "debug console", but if this is anything running in the app, it can ... read the secrets. Regardless of their source.

Or, to put it differently: it matters nothing if your secrets at rest are stored in a quantum-resistent-encrypted vault, secrets.yml or ENV vars, if your logging, backtracing or debug consoles dump them somewhere, you are compromised regardless.

This is absolutely true, but it is far more common for someone to accidentally leave open a display of env vars than a display of files.

I've got this problem to debug in prod, I know, I'll embed the env vars in the source code of the error page. No big deal and it might help me debug that! For a dev used to not using secrets in env vars (for example, when testing locally), this is an easy mistake. He may even use a third party library that embeds features like this without knowing it! Even things like the jmx console can be risky, since access to that may not relate 1:1 with access to secrets.

I've never seen anyone make a similar mistake and accidentally embed all_my_passwords.ini in their output.

I must admit that some of what I've said above has become outdated, since it is now a much more common practice to embed passwords in env vars than it used to be.

Wholeheartedly disagree.

I've yet to come across any git repo that didn't, at some point, commit the all_mypasswords.ini (or, just as bad, the .env.backup). I've only once ever seen a repo that had ENV vatlrs committed (where some moron committed the entire ./dist directory which included a Json that held the copy of ENV at compile time)

My point is not that "one is obviously better", but that none is obviously better. All solutions have downsides. Implementing 12 factor means understanding those.

One could argue that you should never consider any input as “safe to consume”, and if you write defensive app code, would never consider the ENV to ever be secure or what you expect, and would therefor sanitize that input before consuming. If you’re blindly trusting any input just because you’re single tenant, you’re in for some hard to track down bugs and long nights when those circumstances one day change due to random business pivots.

In retrospect, a better name for that section could have been "Separate config from code" as it says in one of the first paragraphs.

How are environment variables insecure?

Envvars have much weaker access controls than files; basically anything in the same PID namespace can read your envvars. Poke around in /proc/*/environ to see for yourself. Files can use user permissions and MAC (AppArmor, SELinux, etc) to secure them against unauthorized processes.

If running in a containerized environment ENV is injected only to the container, at container runtime. If you’re running your containers with a single process, you are doing the exact access control you just outlined. At that point what is the difference between a file on disk with read permissions for the proc owner vs only the proc having the ENV injected to its shell at runtime? I’d maybe even say the ENV at runtime is more secure than the chown’d file on disk because the ENV injected variant is ephemeral and when that container/proc dies, so does the ENV sitting in its memory, whereas the other leaves a file on disk potentially depending on where it was written to, regardless of who can access it. Then there’s the fact that a lot more people than you think run containers as root so all file access control goes out the window.

From a security perspective it is always more secure to have something only in memory of the running process than it is to have a file on disk regardless of file permissions

Containers don't enforce that isolation. Another process can nsenter the container's PID namespace. You also have issues if your container's PID1 creates subprocesses.

File are not necessarily written to disk. e.g. the Secrets CSI Driver loads secrets directly from a secrets store as virtual files within ephemeral volumes.

> the Secrets CSI Driver loads secrets directly from a secrets store as virtual files within ephemeral volumes.

You could totally spin this as agreeing with 12factor… the virtual files are attached resources that are part of a deployment. The configuration, expressed as env vars, configures where to look for those secrets (ie. their filesystem path) and where they’re used (e.g. as a templatized database URL.)

12factor says config should be env vars, it doesn’t say secrets should be. It’s unfortunate that the site lists credentials as an example of configuration, but… it’d be great IMO if we could sidestep a lot of contention by just considering this to be errata of the example, and that secrets are left as an exercise to the user.

The basic idea is still valid. You might just have to use a more appropriate mechanism when using containers. For example, the explanation of the Config factor explains that a config file can also be used.

do you have any examples of more secure ways of injecting secrets?

Check out the Secrets Operator for Kubernetes. Injects your secrets from a secrets store as a file mounted into your container.

And how is /var/run/secrets any more secure than env?

I guess you avoid the risk of accidentally logging secrets with other env variables but otherwise it seems to be just as secure/insecure.

The secrets aren't in a shared location. They're stored in an ephemeral volume specific to each container which other processes cannot access.

If someone has enough privileges to access /proc/*/environ of another process (i.e. root or the same user or process or child process) then they should be easily able to reach inside the container, no?

Other processes running as the same user can be blocked from reading /var/run/secrets via the likes of SELinux/AppArmor whereas they can't be blocked from reading /proc/environ.

It's a pretty fine distinction and I don't know how many people actually bother doing SELinux etc. in practice, but theoretically it's marginally better.

AppArmor can restrict /proc, see example from docker: https://github.com/moby/moby/blob/master/contrib/apparmor/te...

how does the application pick it up? We use the built in secrets that are injected into the container as env vars and then the application picks it up that way. Not trying to sound combative, just looking for better ways to do things.

Read the file on startup.

Bonus: you can watch the file for changes. Which means your app can pick up rotated secrets without a process restart, whereas if you inject secrets via the environment they're fixed for process lifetime.

it's both true

putting any form of secrets into a env var is deeply insecure

using env is the most simple uniformly available way to have configs

through I would take it a step further, anything which must not be leaked probably should not be in any config, weather env or file or similar, it should be injected through other means if viable

Reply on news.ycombinator.com