The Twelve-Factor App (2011) | Hacker News Reader

The Twelve-Factor App (2011)

12factor.net

381 pointsedward254 comments

the twelve-factor app is a set of recommendations from 2011 that are based less on engineering principles and more on the capabilities of Heroku and containerized infrastructure in 2011. For example:

> Another approach to config is the use of config files which are not checked into revision control, such as config/database.yml in Rails. This is a huge improvement over using constants which are checked into the code repo, but still has weaknesses: it’s easy to mistakenly check in a config file to the repo; there is a tendency for config files to be scattered about in different places and different formats, making it hard to see and manage all the config in one place. Further, these formats tend to be language- or framework-specific.

> The twelve-factor app stores config in environment variables (often shortened to env vars or env).

yeah the reason they argue for this is that the people that wrote this worked on Heroku, and the way that Heroku worked is you populated the environment variables from some fields in a web app. If you want your config history tracked in version control, or you do gitops, or you have k8s configmaps, or you want to have your configuration files on a mounted volume ... those things are all broadly fine; they keep the configuration state separate from the app deploy state. This document really confuses the forest with the trees and recommends things based less on actual engineering principles and more on the product capabilities of the corporation that produced it. It is an actively harmful set of guidelines.

It is true that this is somewhat influenced by how Heroku works, but ConfigMaps and GitOps do not meet the same security and usability requirements as Heroku config/env vars.

If you want secure config storage on Kubernetes, you end up using Secrets, which ends up being key-value like env vars anyway. If you want similar security with Git you need a layer of encryption, which breaks diffs and requires additional tooling. This all leads back to why high-level deployment tools like Heroku were created.

> If you want secure config storage on Kubernetes, you end up using Secrets, which ends up being key-value like env vars anyway.

You can load the secret file directly into the app, no need to load it as env vars or keep it strictly as key-value pairs.

> If you want similar security with Git you need a layer of encryption, which breaks diffs and requires additional tooling. This all leads back to why high-level deployment tools like Heroku were created.

You can use tools like ejson[1] or sops[2] to get encrypted files checked into Git that have key level granularity on diffs.

There is definitely a place for higher level abstractions than Kubernetes. Mostly it gives operators a standard platform to build from when teams outgrow the PaaS sandbox.

[1] https://github.com/Shopify/ejson [2] https://github.com/getsops/sops

> You can load the secret file directly into the app

Now you're getting away from the spirit of 12factor and hard-coupling them again. The intent is for the app to consume the secrets but have no knowledge or care where they came from.

Edit: misread as "load the secrets directly into the app." Yeah, this is just env vars but different.

My trick for this with k8s Secrets, when there's some sort of config file that absolutely must be consumed from disk, is to

1. create a Secret that defines the file as a key

2. mount that Secret into the container (.secret.secretName in the volume definition) under an arbitrary conventional path, e.g. /secrets

3. define an env-var that gives the full path to the file; where the fallback when this env-var isn't defined is to find the file in the working directory with a conventional filename.

If your own code uses the file, it can read the path from this env-var; while if the core of your container is some opaque tool that doesn't know about the env-var but does take the file's path on the command-line, then you can wrap the tool in an entrypoint script that reads the env-var and passes it as a command-line arg to the tool.

IMHO, this approach is flexible but principle-of-least-surprise obeying for both development and production deployment.

That sounds an awful lot like a .env file to me.

yup thats exactly what .env is, and tbh that doesn't change anything in the bigger picture

...no? It's the opposite.

A .env file allows you to go from knowing/assuming where a particular file is, to having environment variables.

What I'm describing is that you can go from being passed environment variables directly, to knowing where certain files are; which therefore allows those files to be mounted into a container-image in arbitrary places, rather than encoding brittle assumptions about config-file locations into the binary within the container-image — assumptions that usually get in the way during development, when you're not running the binary inside a container.

https://docs.rs/dotenvy/latest/dotenvy/fn.from_filename.html

You still didn't understand my point. The thing that I'm talking about is using env-vars to point at config files that do not themselves contain env-vars, but rather are opaque app-specific config files, in formats like YAML or whatever-else.

I don't use kubernates. We have JVM servers running on AWS. We load secrets from Secrets Manager. Configs are mostly HOCON files stored in S3. The knowledge of where to find them is configured via env vars.

We use a custom setup in which the config values are loaded from multiple sources actually, so we could put everything into secrets manager, or load them all from env vars or HOCON files.

If we had to set every config value in ECS / lambda using env vars, it would be a major pain in the ass, and error prone.

> We load secrets from Secrets Manager

And how your apps authenticate to Secrets Manager ? Did you ever call `env` on a pod that has IRSA configured ?

This is just a middle step to do exactly the same thing but instead of using envFromSecret you use envFromSM

Not OP but an IAM role with scoped access to secrets. Better again, using secretsFrom in the task definition which injects the secret as an environment variable for you.

IRSA in pods and containers is translated to env vars. That was the point.

That if you dont want to pull in k8s dependencies to the code, at the end every approach ends up as a mounted file or env vars.

> Now you're getting away from the spirit of 12factor

The entire point here is that 12-factor isn't the be-all, end-all, so this is irrelevant.

> The intent is for the app to consume the secrets but have no knowledge or care where they came from.

This doesn't really make sense, and is an impossible requirement. The app will always have to know where configuration or secrets come from. Environment variables is just one option of "knowing where they come from". Choosing file storage is just as valid.

I'm not really kubernetes expert, but don't they come from /my-secrets and the application doesn't need to care how that got mounted?

> somewhat influenced by how Heroku works

The Twelve-Factor App is marketing content produced by the former Founder & CTO of Heroku. It's a great piece of content no doubt, but it should be viewed with that lens.

If it's marketing content then it's not doing a very good job. I have known about the 12factor app principles for a long time, and only found out today that it has ties to Heroku.

Still, Heroku would look impressively good if you wanted to deploy a 12-factor app, won't it?

Not all marketing is that straightforward.

Likewise.

ExternalSecrets with K8s. Put them into SOPS, Vault, your cloud’s secrets handler etc. - then reference them in the app as though they exist. They get loaded at deployment.

Doesn't break diffs, doesn’t directly couple anything.

I'd say 99% of my configs are not secret, at least not from containers running nearby.

I'm fine reading a few passwords and tokens from a secure key-value store. But the rest of my config can be a bit more expressive.

Nobody calls it '12 factor' anymore but we still adhere to a generic set of principles as a result (since 12-factor predated docker and kubernetes in the mainstream).

- Logs as a stream: yep, just write your logs to STDOUT instead of a file. Let the orchestrator deal with reading and storing it.

- Configuration in the environment. Apps tend to pull config from different sources depending on the deploy - maybe a .env locally and a secret store in production.

- Port binding - your app listens on a port and you put nginx or whatever in front of it and set up a reverse proxy. K8S services and ingress do that.

I think the biggest criticism you can lay down on 12 Factor these days is that it's not actually written very well and assumes you know exactly what it is talking about.

It's written very well but the audience is server application programmers in 2011, not someone who has written a dozen services yet never seen a log rotator, "production mode", or an AF_UNIX socket in their whole career.

> - Logs as a stream: yep, just write your logs to STDOUT instead of a file. Let the orchestrator deal with reading and storing it.

This is one that always feels odd to me: what happened to rsyslogd?

Not the responsibility of the container. The host deals with those details.

Still, I haven't seen it in quite some time.

*STDERR

The normal output of a service is a log, and should go to STDOUT. The service can separately output errors to STDERR.

The normal output of a non-service command (eg. grep) sometimes/often goes to STDOUT, which is why that command will write errors to STDERR -- even "info" messages that look like logging.

The normal output of a service is nothing.

dunno if you haven't noticed but people haven't been writing self-daemonizing processes that disappear into the background silently for ::checks watch:: at least a decade now. The practice of writing self-daemonizing processes was always a wart, a reflection of how low our expectations are of the job-control expressivity of cli shells. Grab ten programmers who use bash/zsh every day and say "you have a program that runs in the foreground forever, launch it in the background and close your terminal" and see what they do. You'll find some mixture of nohup, launching it and putting it in the background and disowning it, putting it in a tmux session and then detaching from the session, or deciding they have to put it in a container just to get that behavior, or some other workaround to deal with the fact that launching a process into the background is an ergonomic challenge for a lot of people. The circa-2000 status quo actually had a drawback or two.

> Port binding - your app listens on a port and you put nginx or whatever in front of it

Apache for my personal projects, in front of the docker containers.

Heroku didn’t invent ENV variables nor did it dream up the idea of using them for app config. This has been in use long before Heroku existed. I’ll give Heroku credit for making it common knowledge and spreading the use of these concepts, though.

yeah I never suggested that heroku invented environment variables, and the idea that environment variables weren't common knowledge before heroku came along is ... let's say at best revisionist.

True.

If you remember the world before 12 factor you know that even if motivated by self serving incentives, Heroku making this mainstream has been an overall win for the entire industry and we are generally better and more secure for them having done so. We are still benefitting from all of this today, and Heroku who championed it is making death rattles.

While valid points in many contexts, I've seen many other contexts where following this principle/pattern a/ yields none of its benefits b/ results in very significant business impact.

Here is some obvious case observed many times.

A small and very fast pace team builds a server application having limited to no external dependencies. No db even.

Just a server doing stuff.

It does have some config yes, many many key/value pairs even, a config file ended up with a dozen properties just to control Req per sec limits and such.

Now just to deploy the thing, since of course env defined configurations is where things belong in best practice, the dev team setup a process to peer review those env values and despite all of that, fail to get the app running.

Human errors happen but since more mistakes and release delays were caused by misconfiguration than any other reasons, the keen and bright enginnering team all agreed it would be better to have a config repo. One genius in the group argued that having a config service would be even neater than have the app read from a git repo! The whole got so hyped and of course spend the whole next sprint building and testing that. Brilliant, it worked!

Except that of course outages happened, of course. Outages happen.

Bringing down all environments one by one, hence blocking pretty much everyone. Everything went down because hey, since we made that effort which took for longer than we thought to build a config service and have our app reads from it, one of the dev minds thought it was a genius idea to have the app reload the config every 5 mins at runtime so that if a configuration change was made, no need to even restart anything. Of course error handling wasn't robust and well tested since the boss reminded the team we've got to ship a product by the end of the quarter and solving the universe can maybe wait 5 or 10 years after the IPO.

Team resorted to simply track the config in the repo. Kept the runtime reload of the config and things worked just fine.

Things can be done right and impacts could have been avoided. The example may seem like a twisted case since these were all configurations, not environment specific values. But I've just seen so many occurences of env variables being separated from the application repo costing so much and distracting contributors for no benefit that I have grown very cautious of it.

The intent of this part of 12 Factor comes from the Bad Old Days when devs didn't run their own stuff in production. There was a dev team who could push code and cut releases, and an ops team who would then deploy the releases, and who was paged when things blew up.

If you store the database hostname/credentials, as an example, in the "code" (either literally in the app code, or in a config.ini that lives in the repo and gets baked into a release), then the ops people can't quickly repoint the app at a hot-spare database during an incident: they'll need to edit the repo, push it, cut a new release, etc. If there is a long build step (as for a C++ app, for example), then this isn't really tenable during an outage. If this is in an environment variable, their normal deployment scripts can make the change for them.

I agree that this is probably not necessary anymore if your team is all wearing both hats (dev and ops), and its just as quick to make an edit to the repo and cut/deploy a new release as it would have been for the Ops team to run their deployment script in the old days.

Great you pointing out the presence vs absence of dedicated operation enginners. Also that things have evolved more than a bit in team structures and tooling.

Even when ops are perfectly comfortable with git, they are rarely keen on touching dev repos even if deployment times were tenable. If code changes need peer reviewed, how so configuration values don't get treated at least the same way?

Secrets in clear are obviously a no go, I feel though that encrypted secrets in repos has become reasonably simple with mature tooling in this area. Ansible's fast growth got cut short once people started to hear k8s at every meet ups perhaps.

Secrets in the repo is actually something I've seen for a long time, and there's not much tooling necessary for it beyond just shelling out (yes, I know) to gpg in your deploy scripts. I saw this on CVS-based projects very early in my career, which then fizzled out as the "12 factor style" (secrets in ENV vars) took over.

Can I ask what the modern solution for this is?

Also:

> If code changes need peer reviewed, how so configuration values don't get treated at least the same way?

The easy answer to this is a CODEOWNERS file, with the app code being owned by the devs, and the config being owned by ops. But again, not sure how to do this for secrets without their being plaintext in the repo. Do any code review tools support encrypted content?

"Store config in the environment" does not necessarily mean configs are in environment variables - it only means configuration comes from the hosting environment, not the application itself - you are not adding a "settings.json" file to your machine before starting the app.

This way, the same source deployed in, i.e. your AKS cluster in Azure EU north will take the configuration you set for that cluster while the one you run out of an Docker Swarm of RPi Zero's in an Ikea photo frame (I have one cluster like that, called Gibson), will take the configuration from that cluster.

The page https://12factor.net/config disagrees with you:

> The twelve-factor app stores config in environment variables (often shortened to env vars or env).

I would assume that is the interpretation back in 2011. While the mechanism changed, the principle still holds.

The 12 factor app that stored config in env vars found it trivial to migrate to ConfigMaps. The LOB apps today that expect complex ConfigMap layouts (at worst case, directly from the k8s API) are going to be hell to migrate to whatever comes after k8s.

The invention of dotenv files just makes me chuckle and seems to completely miss the point of not putting them in a config file.

dotenv files are great as a piece of developer ergonomics. And that's fine. I use them for that purpose all the time. I would not willingly consent to using them to manage environment variables in production. But I would also not willingly consent to doing it the way that's best for production during development.

I think that's fine. 12-factor is a set of guidelines about what you do in production. There's a whole universe of things that are smart ideas for development and bad ideas for production. Leaving debug symbols in your binaries, leaving ports for connecting a debugger open on your containers, etc.

It's interesting to me that your philosophy does sharply contradict with another popular notion which is that there is great benefit in unifying production and development environments. Curious what your take on that is.

Loading environment variables from a .env file if it exists is no different between development and production.

You just set the variables at startup in prod, and in the file on development (probably mostly because you want to change them all the time).

The devil's in the details.

There's an enormous difference between "I'm going to use a dev-friendly way to manage and swap out all the different configurations that I might want to plug into the app while I'm testing it, but still plug them in using the same mechanism", which is basically what you get with dotenv files, and, "I'm going to configure with environment variables in production and JSON files in development," which is another thing I often see people doing.

Ops person here. .env files at best should be injected into Developer environment by IDE/Editor. Many IDE/editors have this feature along with Docker. If that's not an option (Thanks JetBrains/Visual Studio), second thing is something like Python-dotenv where library loads .env file at startup, injects them into environment variables and rest of the code is configured to use environment variables. This behavior could even be environment variable controlled and should not error if no .env file is present.

I've read a bunch of tutorials promoting env vars to avoid secrets being accidentally checked into git. Step 1 is usually to create a .env file and add it to your gitignore.

... and if you setup a git project and forget the gitignore

... your secrets get checked into git.

The whole .env file holding all your configs thing is a disaster. People forget to update the template, it has to be passed around, etc.

Configure your secrets in Azure Key Vault (AWS Secrets Manager, GCP-something-I-forget) or whatever the non-cloud Vault software is called.

Devs only set client_id, client_secret, and env defaulted to dev. On startup, the app fetches configs for the secret store. The needed secrets are in source control. If the needed secret is missing from the secret store, there is a clear error on startup.

You are welcome, I just saved you weeks of onboarding, confusion, and devs passing around .envs on Slack.

Hard agree. I think it's easy to get stuck on the mental model that "the configuration value is the secret" (which means the configuration itself needs special handling), and the correct mental model should be "the configuration value is the identifier of the secret", with another system (Azure Key Vault, etc) being responsible for converting the secret-identifier into the secret-value.

My company open sourced the .NET library we use for this: https://github.com/Lokad/Secrets

> You are welcome, I just saved you weeks of onboarding, confusion, and devs passing around .envs on Slack.

Don't get it. At one company I worked, .env files for the DEV environment were open (the file itself was stored on Vault, so only engineers had access to it), and one could only access DEV resources via VPN. So, starting a service using the DEV .env file was rather easy an uncomplicated.

Here is the problem with that. A dev pushes a change that requires a change to the env file. Another dev pulls down the code, but doesn't change the env file. Everything breaks. At really bad companies, DevOps sets env vars from somewhere other than that file passed around by devs - another failure point.

My proposal:

1. Key Vault | HashiCorp Vault | whatever is the source of truth for secrets.

2. When a developer adds a key, they add it to Key Vault and update code to use it.

3. On app startup, code checks the dev's local for 2 env vars ONLY, which allow access to key vault.

4. The app then tries to pull in all the env vars it needs by secret name.

5. If any are missing, it fails.

This keeps devs from having to pass around .env files outside of source control. Btw, Azure (and all other cloud) builds can be configured to pull env vars directly from Key Vault, so there's that benefit too.

There's an optional but very helpful sixth step: expose the loaded configuration (by pushing it to the logs, or as a /_config endpoint). For secrets, obviously don't expose the secret value itself, but do expose the vault address, secret identifier, and which version of the secret was loaded. This helps check that an application isn't using an outdated version of a secret.

"5. If any are missing, it fails."

That's literally all you need to solve the scenario you described with .env files.

The difference is any time the .env config is updated, all engineers have to update their .env file.

So with .env file:

- developer adds to .env

- pushes code

- all developers see failures in dev env due to missing env var

- all developers search slack/email to figure out what went wrong

- all developers update their .env file and things are working again

————- In gp’s comment ———-

- one person works on feature that introduces new config

- failure happens in their local environment

- they update config backend and push code

- all other developers get updates with no changes needed

You can also just supply defaults that can be locally overriden.. Benefit is they are reviewable in the same PR that adds the code depending on them..

You can't keep secrets in code most of the time.

Usually not big secrets.

I do and have done both and a blend. Secrets for external services needed in dev usually in parameter store. Kinda depends on the existing work flow. There is so much to do you should be tackling problems that move the needle and chip away at small stuff, pet peeve stuff in smaller time-boxed increments.

It depends, always it depends.

Exactly, thank you for phrasing it much better than I did!

This is the workflow my company uses. It seems to work ok, but it’s hard to maintain discipline around using this pattern consistently

My company enforces this with two rules. One, deployment always builds and deploys exactly what's in the git repository, so there is no way to include anything else (secrets, configuration...) in a deployment. Two, it's obviously forbidden to commit secrets to the git repository (we enforce this through training and reviews, but it's fairly easy to add pre-commit hooks as well).

I've resulted into a pretty similar setup.

We use SOPS and the encrypted .env file is stored in VCS. The key is stored in 1password with instructions on where to stuff it on your machine. Then you just decrypt the .env file and you're off to the races.

For anyone new to SOPS like I was - https://github.com/getsops/sops

The corporate VPN and proxy setups tend to fail or be slow. This would add non-trivial startup time to any app, and catastrophic startup time to integration tests.

I agree, sops / ejson / gitcrypt or something else that encrypts a file in vcs, then use your cloud or shared secret vault to disperse the decryption key.

As far as I can tell this is the standard way of doing things also in a kubernetes setup: The application gets its configuration from the environment variables.

Then in the case of a kubernetes setup. The application is wrapped inside a mechanism that gets the configuration from some where else.

> or you have k8s configmaps [...] those things are all broadly fine; they keep the configuration state separate from the app deploy state

Why are they fine? This seems to be an argument grounded in "it's a newer norm therefore it's better" but I don't see the justification. I've never used Heroku but ConfigMaps are a bane to manage. Even as a general programming principle within application code, colocality is a well-known, long-discussed topic around improving maintainability. If you're not using monorepos, that's currently not a reasonable option with current orchestration conventions.

Even if you are using monorepos, you're highly likely to either get into writing & automating DSLs to feed into your configmaps or do weird things like using Jinja to do file includes in your mounts. That's all before you run into the etcd 1MB size limit & realise that actually this whole system is a hack.

TL;DR newer ain't intrinsically better

History repeats itself. Netlify tried this with the MACH Alliance.

If it is truly harmful, it shouldn’t be sitting at the top of Hackernews. Let’s flag it.

I... don't think the "flag" functionality is intended to communicate "this is bad advice".

It is “harmful” advice, not bad. Regardless, why should it sit on the front page of hackernews?

it's really hard for me to express just how much content hits the front page of HN that, if you follow it, will make you worse at programming.

Because there is productive discussion about it in the comments?

i got a ton of value out of reading the comments though

I feel like each of this points can be reasonably disputed.

1. One app - one repo. There is nothing fundamentally wrong with multiple apps (as in separate pods/containers/processes) being developed in a single repo, as long as they are tightly coupled functionally, share the release cycle, but still need to be deployed separately to get advantages of separate processes and independent scaling. The first example that comes to mind is separation of public api and worker processes, e.g. ruby's sidekiq, python's celery or a generic kafka consumer.

2. > A twelve-factor app never relies on implicit existence of system-wide packages.

This is incredibly hard to achieve in practice if you're not running something nix, and even then there are leaks of dependencies on kernel syscall APIs. Every Rust app implicitly depends on a glibc (except musl ones), which is present in major Linux distros except slim ones such as Alpine. In fact, the whole Docker crutch IMO is needed specifically to alleviate this problem.

3. > The twelve-factor app stores config in environment variables

It seems to me much more brittle, as it forces you to store secrets in envs, which could be less secure, and to abandon structured file configs, which may enjoy type safety, IDE autocomplete and automatic parsing. With envs you are forced to implement parses for non-trivial and non-string config vars yourself. In fact, envs are often committed into repos as well in form of dot-env files, so the point about commit-safety is moot as well.

> This is incredibly hard to achieve in practice if you're not running something nix, and even then there are leaks of dependencies on kernel syscall APIs.

You're misinterpreting that guideline. The text says (my emphasis):

> Most programming languages offer a packaging system for distributing support libraries, such as CPAN for Perl or Rubygems for Ruby. Libraries installed through a packaging system can be installed system-wide (known as “site packages”)

It's not saying not to depend on the operating system (of which glibc is part), it's saying not to require particular language package manager packages to be installed on the machine.

Agreed that the other ones you mention are silly, though. Especially the one about environment variables - that isn't actually reasoned advice, it was just the authors writing down what was most common in Ruby development at the time, on the assumption that that must be the best thing to do.

In my experience, sourcing config from env variables seems like the best thing to do because you can completely ignore where the app is running in favor of injecting whatever you want in any environment of your choosing and your app no longer needs to care about that. The implication is that business logic must not include the content of your config or where it is sourced from. It must simply use the values in the config. This way my apps could give less of a fart if I run it as a container in kube, in nomad, in local on localhost, in a devcontainer, in a directory with vars loaded using direnv, in vercel/netlify and so on. This also allows incredible flexibility in how that config is sourced as long as you simply dump them into the env vars. Want to load secrets from a .env file on local? Sure. Want to load secrets from Vault on production? Run a simple script to load vault secrets onto the env and run your app.

yeah this is the actual point - it makes the code/app totally portable & decoupled from config. I find it significantly better than having to have the app code choose different config based on interrogating facts about its runtime (eg am i in prod? use this value. am i in test? use this)

Everything you both wrote is spot on - but there's nothing about it which means you have to use environment variables! All of those mechanisms could be used with any other means of delivering the config at the point of process invocation - a config file, command line arguments, YAML to standard input, whatever.

Personally, i would pick a config file over environment variables any day: easier to manually inspect and tweak, easier to give structure to, doesn't get implicitly passed down to subprocesses, can be secured with familiar file permissions, etc.

Problem with config files is you are forced to process it properly in pipeline so it can be delivered during deploy AND depending on your runtime environment, you might have to redeploy to change things.

Environment variables exist on just about EVERY runtime, changed without requiring a pipeline run.

However, subprocess is interesting but I've never run into subprocess that is not sharing a config. Where I worked, if you have subprocess that can't see those configs, you create a new service.

I'm not sure what you're saying about pipelines. Ultimately, every app process is started by some manager. It's just as easy for that manager to create a config file as it is to set environment variables.

If your point is that existing managers support setting environment variables but not creating config files, then sure. But that isn't a fundamental truth, it's just working with the tools you have.

Say you have a GitHub Action that pushes a container with your app and deploys to a Kubernetes cluster. Where is your config file going in this step? How do you inject that YAML/HCL/JSON file into the app to configure your app?

It just seems far easier to have your kube config inject the env variables from your kube file and that way you don't have to worry during deployment how to add that file and you don't check it into the source. This applies to basically any way you deploy your app, especially containerised ones. Unless you're running your app in an EC2 machine where you can go in and easily edit the file (not that you can't do it in a container), it just seems like extra steps. I've found the happy medium to be sourcing default configs from a file that can all be overridden through env variables

Makes sense. But this is exactly what i was saying: the tooling you're using makes it easy to set environment variables, and doesn't make it easy to add config files, so in this case, environment variables are easier to use than config files.

But there's no fundamental reason for that to be the case. You could imagine a very nearby parallel universe where Kubernetes config files don't let you set environment variables, but do let you inject files. In this universe, you can see other deployment approaches that make config files easier (Puppet and suchlike). And if you're building something from scratch, you can make the choice.

Well... yeah but in the universe we live in, env variables are the most convenient, interoperable way of configuring apps and work on almost every platform. I guess I could work on a way to sit and dump config files into my apps but what's the point when env vars work well and are convenient to use right now?

I heavily agree. I used to work on a LAMP stack where the deployment was just dumping the code into an EC2 instance. All is fine until there's a weird system dependency error that isn't documented because the AMI being used has just been hand built over the last three years. Have fun tracking down that specific PHP system dependency when AL2 goes EOL. I don't want to write code that doesn't have _documented_ dependencies on system packages. Even just a Dockerfile serves as solid documentation on what we're using.

Well, glibc could be considered part of the operating system, although this too is arguably not correct, because some technologies go out of the way to not depend on anything present in the system and be completely self-contained, e.g. fully statically linked Go binary with no Cgo usage.

Nevertheless, glibc is not the only system dependency which is often used by apps, another such example is JVM of a specific version, which is not tracked by any of the popular JVM build tools and package managers; and OpenSSL, which is too installed by the system package manager.

Dockerfiles exists to fix this by defining all the necessary installation steps as a sequence of instructions, but this is far from a proper package manager.

The libc is absolutely part of the operating system. Some people and tools go out of their way to avoid using it, but they are completely mistaken to do so. Go in particular has wasted a huge amount of effort on this absolutely pointless exercise.

And not just libc - on a typical unix, there are a small number of other libraries which constitute part of the operating system, like libm, libdl, and libpthread. They are part of the means by which the operating system delivers the POSIX environment.

You're right that the JVM is a dependency, and that there is very poor versioning support for it in production. Everywhere i've worked, we've come up with ad-hoc solutions for it, and none of them have been very good.

> There is nothing fundamentally wrong with multiple apps (as in separate pods/containers/processes) being developed in a single repo, as long as they are tightly coupled functionally, share the release cycle, but still need to be deployed separately to get advantages of separate processes and independent scaling.

If they're deployed separately, they don't share the release cycle. Sooner or later you'll end up with mismatched versions of them running (e.g. perhaps one piece fails to deploy). So you need to be prepared for that, so you want them in separate repos so you can easily test with different versions.

> This is incredibly hard to achieve in practice if you're not running something nix, and even then there are leaks of dependencies on kernel syscall APIs. Every Rust app implicitly depends on a glibc (except musl ones), which is present in major Linux distros except slim ones such as Alpine.

Most higher-level languages avoid depending on a specific glibc; as long as you have a working runtime for that language then your app will work with it. And sure in some cases you resort to docker or something. It can indeed be hard but that doesn't mean it's not worth doing.

> If they're deployed separately, they don't share the release cycle. Sooner or later you'll end up with mismatched versions of them running (e.g. perhaps one piece fails to deploy). So you need to be prepared for that, so you want them in separate repos so you can easily test with different versions.

In that case I'll just rollback the first one. I see no reason to logically decouple two processes into two services when the separation is technically driven. They share the queues, the database, the domain, further division will just make things more complex with no obvious advantage.

> In that case I'll just rollback the first one

At that point you're not deploying separately any more. They explicitly say you can have multiple processes and process types as part of the same app (see #8). The point is that "repo of its own" should align with "atomic unit of release/deployment".

We disagree on fundamentals then and I think more of the world agrees with me and with the 12 factors.

I agree with your view of the first rule and would gladly store multiple related apps in one repo. No rationale for why one repo should only containing one app is given, therefore that rule can be disregarded.

Dynamic or managed languages can often ignore most of the hassle with system packages.

> 1. One app - one repo.

There is a U-shaped utility to monorepos. Most projects fall in the middle somewhere, largely ones managed by independent engineering teams that have no centralized platform/devops/whatever-we-are-calling-it-these-days team.

Point 1 was written when it was standard practice to stick all of your organisations code in a single repository. The concept of splitting things (history, diffs, etc.) up per app was foreign to a lot of people.

They also say every server should be a separate process, which is debatable. When using languages that lack real threads (Ruby, Python, etc) you have no choice.

"One app - one repo"

Where did you see that being expressed?

https://12factor.net/codebase

It says there that a codebase can contain multiple repos. They don’t mean a monorepo when they say codebase

> A codebase is any single repo (in a centralized revision control system like Subversion), or any set of repos who share a root commit (in a decentralized revision control system like Git).

They do mean that a repo = a codebase. A remark about "set of repos" is just a technicality of distributed VCSs, meaning that each copy cloned from a single source (or from other copies) is a single repo.

I like this in general, but the number of times I've had to delay a release because some non/psuedo-technical person has pulled out "12 factor" like a universal yellow card has been enough for me, overall, to ignore it almost completely. I guess the same has happened to me for "agile" honestly. I recognize the intent of guidelines like these, but they seem to be much more useful to people whose only "value" is to provide ivory tower technical leadership.

> some non/psuedo-technical person has pulled out "12 factor" like a universal yellow card has been enough for me, overall, to ignore it almost completely.

Ignoring it completely on principal is just as bad as the people who treat it like mandatory dogma.

I’ve also had over-eager junior engineers or wannabe architects use the 12-factor article as mandatory requirements for any launch. It takes a firm and unified explanation that these are good goals to strive for, but in the real world we need to make compromises for launch and choose which aspects to delay or defer.

Ignoring it completely is a viable strategy. A lot of 12 factor sounds like common sense to people who regularly ship stuff. For example, if I fail to put the config in the environment, the risk that introduces will be immediately obvious in the PR and someone would be able to raise that concern, with real justification, not because it's declared in a 12 factor manifesto.

I think ignoring it completely is a viable strategy once you've already gone through the stages - when your mind is in the right place.

I have moved on a long way from the days that 12 factor was my bible, but those days completely redefined my way of developing software. I'm not exaggerating here - it completely shifted my approach.

I guess it's similar to Scrum in that way - it isn't a great end point, but if you're stuck in the old ways (i.e. the methods used before it was even a thing) then it's at least a good approach to sink your teeth into, try it out, shape your approach, and move on from there.

Basically, I'd rather see a replacement that improves on 12 factor than throwing out the book entirely.

> A lot of 12 factor sounds like common sense to people who regularly ship stuff.

In 2023, yes. A lot of the 12 factor methodologies became table stakes in the modern cloud era. Back when this was written "cloud" was not nearly as ubiquitous as it is now, although we were certainly headed in that direction. A lot of the comments here are viewing this through a 2023 lens and forgetting it was written in 2011.

I call that weaponized dogmas. It’s at least better than people who insist on being DRY by writing an inscrutable generic helper function which will only save 3 or 4 lines of code duplication.

Nothing I hate more than a useless "helper" function that wraps a built-in feature of the language and gets buried in a "common lib" package that you need to inspect when you want to understand what is going on, instead of just relying on your knowledge of the language. Do we really need a function to map a dictionary and a list of keys to the corresponding values?

I'm a huge hater of these and I think they almost always decrease readability of the code and slow down the development and review process.

I want to read and write the language I'm developing with, not the helper methods of some overeager developer who wanted to leave his mark on the codebase.

Oh my god I got a flashback to Groovy's missing method [0] which caused us so many issues, and made our codebase so unreadable.

When you discover it for the first time, you think, "huh, neat, why not use it". Before you know it, nobody can reason about your code and if you leave, nobody wants to maintain it (esp. if your test suite is not extensive).

[0] https://groovy-lang.org/metaprogramming.html

The absolute counter to weaponized dogma is itself weaponized dogma. There are times when implementing DRY, 12FA make sense and times when it doesn't. Be careful with hard-lined stances.

Give an example. Like most things, it's context dependent and there are grey areas, but I think most devs use 12 factor like a north star. I wouldn't block a release due to some minor deviation, but generally lacking alignment should be treated as tech debt as a minimum. If a release regresses significantly in one of the factors, I think it's fair to block or at least force more detailed review to understand why the dev felt the tradeoff was worth it for that release.

There's the "Plumber Problem[1]" by John Siracusa where you watch a story and see a problem in your specific domain that leads you to mistrust the whole story.

With the 12 factor nonsense, I see the guidance of "Dump unformatted logs into stdout and make it someone else's problem; they'll hadoop or splunk it and everyone wins." There is no magic splunk that absolve you (the developer) of generating useful data to logs, if you expect useful data out of your logs. You can't reverse "cow -> burger -> sewage" and shouldn't expect some magic hadoop to reverse entropy without a ton of work.

So -- I guess the 12 factor app dictum is fine for some trivial "I rewrote twitter in 80 lines of erlang" app, but in my opinion it is a "if we can land a man on the moon we can land a man on the sun" misstatement of complexity in actually getting things done in the real world.

[1] https://hypercritical.co/2023/08/18/the-plumber-problem

Spoken like someone who has never had to work on a Java application with 22 custom log files, not including stdout, stderr, and two separate syslogs.

Yes, if you want useful data from logs you'll need to log useful data, but this is a case where 12 factor is so successful / dominant today I don't think you even know what it's arguing against anymore.

> this is a case where 12 factor is so successful / dominant today I don't think you even know what it's arguing against anymore

Thank you! This is the same thing people miss when they read things like the Agile Manifesto, or about DevOps, or even architectural choices like React's original pitch for one-way data binding - or, dare I say, microservices.

You can read these things today, and see flaws - holes in the argument, mistaken assumptions, or misread them as claiming to be superior to things that have been developed since. No - to understand what they are talking about you need to know what was current common knowledge or accepted wisdom in the world when they were written.

No, writing files is nearly as bad as writing to stdout -- both imply someone else has to parse the output.

If the logs matter -- write them to an API that retains log structure by serializing it.

If the logs don't matter, then they don't matter, but typically the logs are used for all sorts of analytics processes after the fact.

Asking someone else to parse your data is rude.

Again, "this is a case where 12 factor is so successful / dominant today I don't think you even know what it's arguing against anymore"

The twelve factor app says: A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout. During local development, the developer will view this stream in the foreground of their terminal to observe the app’s behavior.

That guidance is deeply foolish; I have to assume the rest of the guidance is just as silly for anything but a toy application.

Where does it say "unformatted"? All it says is that your logs should be sent to stdout.

You write your logs in whatever structure you want and have something in the runtime environment (k8s, systemd, file redirection in bash whatever) forward them to the eventual storage location. I don't see the issue.

So -- there's a file, that's created by some something in a container emitting a string of bytes. A file must go through a parsing and serialization phase to be useful to a logging infrastructure. Just because you decorate the contents of that file with { and } and other punctuation doesn't make that file a string of json records, until something's read and verified it.

I'm supposed to just load it into memory hoping it's json or avro or whatever? Nope, errors happen; there's a limit to the number of bytes that can be sent from one place to another before the kernel may interrupt it (you're writing an application log and your runtime emits it's own debugging message; they get intermingled because they're both being written to a fifo).

Maybe this is fine (usually it's fine) but eventually it isn't fine. Use a logging library, categories your logs, maybe start off with stdout but eventually put on some big person pants and use a logging api and use the stderr/stdout for high cardinality exceptions, not "all my logs"

Just so I understand, you are saying that the entire concept of the 12 factor app is bogus because it tells you to write unformatted logs to stdout because anything you write to stdout is by its very nature unformatted because even if you're outputting formatted data something else in your program might decide to concurrently write into your stdout buffer while you're writing a log line and that will corrupt your entire data stream irreparably?

I'm saying that, the one thing I know about looks like obvious nonsense -- "write your logs to stdout and let someone else sort it out" -- so I assume the rest of it is equally facile and silly.

So now, when I see a reference to "12 factor app" I assume "ah, a hello world app from someone who's going to leave and let someone else figure out how to make it work in the real world at real loads with real regulatory or compliance needs"

In other words, for me, it's a shibboleth for "toy" or "unserious piece of shit"

There may, in fact, be parts of the guidance that aren't deeply incorrect. But anyone who points at it as a way to do things, without caveats, is suspect.

It's not 'someone else'. It's whatever environment the application ends up running in.

The idea of 12 factor is to define a reasonable interface for your application. The environment doesn't need to worry about providing interfaces to logging services or collecting files from a custom directory, or making sure the application isn't going to fill the disk with log output - if the app is 12 factor, the only thing the environment needs to do is collect stdout.

As a 12 factor app, write such that what you put on stdout is useful for that purpose.

As a system hosting 12 factor apps, collect stdout and put it somewhere useful.

As a developer of a 12 factor app, when you want to see your log output, go wherever the environment puts it and look there.

That's a useful contract that works at scale. Not sure why you think it doesn't.

That's a fascinating way to live.

I'd love to know what other things you saw one part of and then dismissed fully without further investigation.

Are iPhones unacceptable because they suggest you set a 4 digit pin for access? Aeroplane safety demonstrations? Languages with gendered verbs?

12 factor does not say ‘don’t use a logging library’

Use a logging library. Configure it to output well formed and categorized log messages to stdout. Let the runtime environment take that stream of data from there.

It certainly doesn't say "use a logging library"

it says: A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout. During local development, the developer will view this stream in the foreground of their terminal to observe the app’s behavior.

In staging or production deploys, each process’ stream will be captured by the execution environment, collated together with all other streams from the app, and routed to one or more final destinations for viewing and long-term archival. These archival destinations are not visible to or configurable by the app, and instead are completely managed by the execution environment. Open-source log routers (such as Logplex and Fluentd) are available for this purpose.

The event stream for an app can be routed to a file, or watched via realtime tail in a terminal. Most significantly, the stream can be sent to a log indexing and analysis system such as Splunk, or a general-purpose data warehousing system such as Hadoop/Hive. These systems allow for great power and flexibility for introspecting an app’s behavior over time, including:

This is, at scale, utter horseshit. If it said "do what's expedient, but use a library and be flexible in how you generate logs, but for goodness sake use any of the publicly available libraries to be expedient now and reliable and flexible later" I'd shrug and move on, but it's just absolute garbage. "Do a shitty job and someone else will fix it for you" is not good advice, even if it lets you get to market quickly.

It's clear you haven't actually taken the time to understand what 12 factor is saying, and you're assuming that anything left unsaid is an implied permission to just screw everything up.

A more reasonable reading of 12 factor is to see that since it suggests that the output streams should be gathered up from multiple running instances of the application, and collated together in a system like splunk, that it behooves you as a 12 factor app developer to build your app such that your output stream is amenable to that approach.

I.e., that your output is a stream of structured serialized data.

You haven't clarified what you think is a better practice than this.

I've had the pleasure of dealing with devs who read this and decide it says "emit json formatted text to stdout and let someone else figure it out, just like it says in some web page." The guidance is explicitly "send it to stdout and let someone else figure it out." Which is terrible advice.

The words don't suggest anything else. Maybe your experience suggests maybe you should use a library and reserve the ability to serialize that data to a proper api if necessary, but that's you not the words in the 12 factor app, and some large subset of people just do the minimum necessary to get the ticket off their queue, and if that means "printf log" that's what this advice in this suggests they'll do. It's splunk's problem now!

And my advice is "use a damned logging library; understand the distinction between using a networking stack and API that manages abstractions like event size and retrying and such and stdout, which offers none of that."

Sir, this is a Wendy's.

> It certainly doesn't say "use a logging library"

Right. It's not telling you anything about whether to use a library or not. All it's talking about is where you output your logging streams to, not about how you produce or format those streams. Is it bad advice because it doesn't tell you you should wear sunscreen when you go outside in summer?

(FWIW I haven't seen any logging library add enough value to be worthwhile. Eventually graduating to structured event logs is a good idea, but having started with a logging library doesn't actually get you any closer to that).

What 12 factor is telling you to do is to not build a log file rotator into your application.

It is the least controversial and most broadly beneficial of the 12 factors.

Why do you think it’s foolish?

It says "send it to stdout and let someone else figure it out."

As that someone else, who often has bug reports like "why aren't my log events all in one record in my thing? Because you filled your stdout with newlines!" "Why did my message vanish? It was important!" "because it's malformed json because your runtime had an error; it's over there instead of over here."

Programming's a stack of details; all of which matter. Log using a logging API where available; use stdout/stderr for exception logging only, if at all possible. If you want your logging to be useful, think carefully about how and what you log.

That’s a total misreading of what 12 factor is about.

All it means is to let someone else (other than the program itself) figure out how to get the log messages (whatever you decide they should be) from your process (running wherever it is) to wherever you want to have those logs for inspection.

Do not hardcode into your application the assumption that they are written to a file or sent to a port.

In your application, write them to stdout.

In local dev, stdout will probably go to an interactive console. In a container running in a cloud hosted cluster it will probably go to a networked log aggregator that collects it all into an indexed time oriented data store along with other logs from other running instances.

Because you used stdout, that’s easy to do.

If you wrote them to a file whose name your application manages and changes itself to manage log rotation… either of those is hard to do.

The reason people pay outrageous prices for Splunk is you can get some pretty decent burgers out of sewage if you know what you are doing, and from personal experience the results can be totally game-changing if you are stuck with a mishmash of different log formats and have to support a large distributed application.

> Give an example.

I can give an example. The part about sticky sessions is good generic advice for someone building generic web services, but there are situations where sticky session-like functionality or sharding-like behavior might be a better choice. Anyone adhering literally to the 12-factor guide would get hung up on trying to move that data to redis or memcached:

> Some web systems rely on “sticky sessions” – that is, caching user session data in memory of the app’s process and expecting future requests from the same visitor to be routed to the same process. Sticky sessions are a violation of twelve-factor and should never be used or relied upon. Session state data is a good candidate for a datastore that offers time-expiration, such as Memcached or Redis.

The guide also has a strong preference for languages like Ruby, Perl, and Python. There’s even a quip about how 12-factor prefers languages with a REPL, which in the wrong hands becomes an argument against using Rust or other languages.

People who think the entire world is simple CRUD web apps tend to think that guides like this are completely universal. It’s a good guide for people making common web apps, but it’s not a perfect methodology for every backend service.

> People who think the entire world is simple CRUD web apps tend to think that guides like this are completely universal

That's the very first point they make:

> In the modern era, software is commonly delivered as a service: called web apps, or software-as-a-service. The twelve-factor app is a methodology for building software-as-a-service apps that...

Anything not web app/SaaS they express no opinion on.

I think it depends on how you interpret "rely". If your app can gracefully recover to a reasonable state from a failure to resume the expected session, then you could argue it's not that reliant on it. Many systems use sticky sessions as simply as an optimization, which is fine.

You didn't actually give an complete example. When are sticky sessions a better choice than persistence?

No amount of dogma should interfere with a deployment within the scope of an MVP. Once it's out of MVP then anything not addressed best practice-wise is tech debt. If your organization had adopted 12FA as a set of best practices then those should be met, but they should never interfere with a deployment.

12FA is not a single box that is to be checked. It can and in most cases probably should be implemented over time as the product matures, taking each point, point by point (and maybe even breaking those up) and adding that into the product. If you've done good engineering up front (abstractions and interfaces where it makes sense, not hard coding everything) this should not be an issue.

inb4 YAGNI: this is just as abused as 12FA.

EDIT: the biggest thing 12FA is missing are concrete examples that junior engineers can reference.

I've had issues like that before. The solution was to have a clear, documented process, and point them to that.

I've learned to take issues like that in good faith. Why are they raising an issue? Is the existing process not clear? Is there a deficiency in it? Is there a lack of reliability? If reasonable concerns are found, the process can change (and the documentation is to be updated).

> The solution was to have a clear, documented process, and point them to that.

This has its own drawbacks. I worked for a company, in which Process Was God, and Thou Shalt Do Only That Which God Has Proclaimed Good.

It worked most of the time, but made it very, very difficult to introduce flexibility and orthogonal approaches.

There's really no way to duplicate the results of experienced, cohesive teams of high-functioning engineers, with process. We can get similar results, in many ways, but there's always a price to be paid.

T. A. N. S. T. A. A. F. L.

Well, if your app log directly to files on disk (extra bonus if it doesn't respect any verbosity setting) or will only works if secrets are hard coded in config files (even if this could be work-arounded at deploy time probably), I for one will do all that I can to avoid your app reaches production.

Yeah, I work in ops^W platform.

This is more a reflection of the organizational structure that allows or tolerates such behavior rather than anything to do with the 12 factor app methodology. I think most reasonable people would agree that if you were able to incorporate most of the list, you’re doing alright.

> but they seem to be much more useful to people whose only "value" is to provide ivory tower technical leadership.

I've found it pretty useful in practice. I don't see why it's much more useful to those people than it has been for me.

TBH, most of the value in things like this is for consultants who go into shops that really don't know better.

Being able to point management at this can be really useful.

For most of us here, it is 99% common sense.

Right?

There are gold nuggets in many of this style of "manifesto". It just takes a handful of people treating it as dogma to stink them up.

Indeed.

Hard to believe that treating best practices as strict rules would not turn out to be best practice, isn't it?

Twelve factor app: use environment for config

Docker: Don't use environment for config, it's insecure

I like and use many of the patterns of 12 factor, but indeed some of these were written in the context of VPS, where environment is stable, secure and more constant (unlike containers, where environment might end up shipped in one of the layers). In the age of containers, this particular point has been a bit of a struggle for me. Docker secrets don't always play well, and you need to do some gymnastics to make it work.

Could you elaborate? How are environment variables insecure? If the secret will be present in the application throughout it's lifetime, then what's a better alternative?

Just because a secret is injected into the application as an environment variable does not mean it's handled insecurely elsewhere. For example, AWS ECS containers have built-in support for fetching secrets from Secret Manager at startup and passing them as environment variables: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...

This fetches the secret at the time the container starts up (IIRC), and uses the IAM credentials of the running application to fetch them from Secret Manager (so it must have permission to the secret).

The merit of using environment variables seems to depend entirely on how they are, well, configured. With a mechanism like this I don't see much disadvantage. (The main disadvantage I see is that generic malware that attempts to dump environment variables might capture them, but defending against that threat is largely obfuscation, unless you aren't storing the secret at all permanently in memory.)

- any sub process has full unlimited access to env variables

- env variables are prone to be dumped in various contexts and might leak secrets to e.g. logs

- env variables might leak, e.g. in context of docker build stages etc. (but then using building a docker image to build both the image and the software is often anyway not a good idea)

- env variables are often easily accessible by other processes

the method you mentioned of ad-hoc injecting a secret as env by some form of security manager for containers avoid many such problems

> - env variables are often easily accessible by other processes

A process would need to have the same permission (same user) or root to read /proc/$PID/environ no? or what mechanism is there for a process to read another process env vars (not children processes, that was already mentioned)? unless of course, a process dumped them in some world-readable file (like logs as mentioned) or some other silly way

Exactly. The operating system already offers security for environment variables: use it. And you get the benefit of increased security in other places by not sharing users between applications.

except no it does not, env variables wherent designed to be secure and the degree of secruity the system provides is like a bad joke if you take security serious

still works if your system is a single application container

otherwise it does not

I'm not sure I agree. If that were true, the security of the user system itself would also be a bad joke and it's not. Regardless, what I meant is that the OP made it sound like they're using the same server and user and container for multiple applications, but if they did things correctly, the risk of sharing environment variables across applications wouldn't exist.

The problem is the env stays readily accessible to most programs with similar permissions and all child programs if no special actions are taken.

This is (part of) the reason programs accept secrets like passwords normally only (or strongly recommended) over stdin or similar instead of env variable or cmdline.

Or why it's not rare (or was in the past, probably is today) to have a encrypted cert file and inject the password to it through a side channel.

It's also not rare for programs to dump environment variables in various situations, including to logs. Or e.g. an intrusion detection program might snapshot all programs running + their cmd arguments + their (creation) env and then run analysis on it.

And while you can use e.g. selinux and similar to add a ton of security and prevent such issues, or use systemd to run the program under ad-hoc users with minimal permissions and various isolation it's very often not done.

EDIT:

To be clear I'm not saying you can't use secrets in environment variables safely.

I'm saying by default by their design environment variables are not "secure for secret passing". But many things for which stuff like that is true can, under the right circumstances, still be used securely.

> env variables might leak, e.g. in context of docker build stages etc. (but then using building a docker image to build both the image and the software is often anyway not a good idea)

Since the rule describes runtime configuration, it shouldn't be involved with the build process at all

Presumably because /proc/$PID/environ exists and can be read by the user that owns the process — which means that any attacker that can figure out how to spawn an arbitrary subprocess running as the daemon user (a pretty common vulnerability class!) can exfiltrate all your env-vars.

It's similar to the reason that you're not supposed to supply passwords on the command-line (/proc/$PID/cmdline exists), but instead either read them from stdin, or do the big dance of

1. creating a file that contains the password, is owned by root, and is 0400;

2. having your daemon start off running as root for just long enough to read the password into process memory;

3. then having your daemon drop privileges before doing any work, so that someone exploiting the daemon won't have the privilege as the "do the work" user to read the file with the password in it.

My comment is specific to docker, although environment as a broader sense can be discussed.

I remember reading that it is possible to end up in situations where your environments can end up in the layers of the docker image. Also, you can read the secrets of a container with inspect, and apparently linked containers might be able to access other containers's secrets.

I think it was this article: https://snyk.io/blog/keeping-docker-secrets-secure/

On the other hand, without platform support there are not a lot of easy alternatives. It at least encourages separation of config and code, keeping secrets out of vcs.

Agree. The main benefit of environment variables is portability. They work pretty much the same way in every platform I have encountered. As soon as a system gets into files, it's possible to get bogged down in the complexity of Windows vs. Unix, file permissions, SELinux, etc. The simplicity is often worthwhile over the extra security theoretically possible with well-managed files.

Yeah, I've never been a fan of secrets in env vars for this reason. It is shocking how frequent dumb debug consoles have accidentally exposed env vars. You really don't want that to accidentally expose your private keys.

There's really nothing wrong with secrets mounted on the file system via k8s. But, of course, all of this depends on your deployment environment.

Sometimes env vars are the least bad option, because secrets really are always hard.

> dumb debug consoles have accidentally exposed env vars

Huh? If the app can read the secret, then the app can... read the secret. Maybe I'm misunderstanding what you call a "debug console", but if this is anything running in the app, it can ... read the secrets. Regardless of their source.

Or, to put it differently: it matters nothing if your secrets at rest are stored in a quantum-resistent-encrypted vault, secrets.yml or ENV vars, if your logging, backtracing or debug consoles dump them somewhere, you are compromised regardless.

This is absolutely true, but it is far more common for someone to accidentally leave open a display of env vars than a display of files.

I've got this problem to debug in prod, I know, I'll embed the env vars in the source code of the error page. No big deal and it might help me debug that! For a dev used to not using secrets in env vars (for example, when testing locally), this is an easy mistake. He may even use a third party library that embeds features like this without knowing it! Even things like the jmx console can be risky, since access to that may not relate 1:1 with access to secrets.

I've never seen anyone make a similar mistake and accidentally embed all_my_passwords.ini in their output.

I must admit that some of what I've said above has become outdated, since it is now a much more common practice to embed passwords in env vars than it used to be.

Wholeheartedly disagree.

I've yet to come across any git repo that didn't, at some point, commit the all_mypasswords.ini (or, just as bad, the .env.backup). I've only once ever seen a repo that had ENV vatlrs committed (where some moron committed the entire ./dist directory which included a Json that held the copy of ENV at compile time)

My point is not that "one is obviously better", but that none is obviously better. All solutions have downsides. Implementing 12 factor means understanding those.

One could argue that you should never consider any input as “safe to consume”, and if you write defensive app code, would never consider the ENV to ever be secure or what you expect, and would therefor sanitize that input before consuming. If you’re blindly trusting any input just because you’re single tenant, you’re in for some hard to track down bugs and long nights when those circumstances one day change due to random business pivots.

In retrospect, a better name for that section could have been "Separate config from code" as it says in one of the first paragraphs.

How are environment variables insecure?

Envvars have much weaker access controls than files; basically anything in the same PID namespace can read your envvars. Poke around in /proc/*/environ to see for yourself. Files can use user permissions and MAC (AppArmor, SELinux, etc) to secure them against unauthorized processes.

If running in a containerized environment ENV is injected only to the container, at container runtime. If you’re running your containers with a single process, you are doing the exact access control you just outlined. At that point what is the difference between a file on disk with read permissions for the proc owner vs only the proc having the ENV injected to its shell at runtime? I’d maybe even say the ENV at runtime is more secure than the chown’d file on disk because the ENV injected variant is ephemeral and when that container/proc dies, so does the ENV sitting in its memory, whereas the other leaves a file on disk potentially depending on where it was written to, regardless of who can access it. Then there’s the fact that a lot more people than you think run containers as root so all file access control goes out the window.

From a security perspective it is always more secure to have something only in memory of the running process than it is to have a file on disk regardless of file permissions

Containers don't enforce that isolation. Another process can nsenter the container's PID namespace. You also have issues if your container's PID1 creates subprocesses.

File are not necessarily written to disk. e.g. the Secrets CSI Driver loads secrets directly from a secrets store as virtual files within ephemeral volumes.

> the Secrets CSI Driver loads secrets directly from a secrets store as virtual files within ephemeral volumes.

You could totally spin this as agreeing with 12factor… the virtual files are attached resources that are part of a deployment. The configuration, expressed as env vars, configures where to look for those secrets (ie. their filesystem path) and where they’re used (e.g. as a templatized database URL.)

12factor says config should be env vars, it doesn’t say secrets should be. It’s unfortunate that the site lists credentials as an example of configuration, but… it’d be great IMO if we could sidestep a lot of contention by just considering this to be errata of the example, and that secrets are left as an exercise to the user.

The basic idea is still valid. You might just have to use a more appropriate mechanism when using containers. For example, the explanation of the Config factor explains that a config file can also be used.

do you have any examples of more secure ways of injecting secrets?

Check out the Secrets Operator for Kubernetes. Injects your secrets from a secrets store as a file mounted into your container.

And how is /var/run/secrets any more secure than env?

I guess you avoid the risk of accidentally logging secrets with other env variables but otherwise it seems to be just as secure/insecure.

The secrets aren't in a shared location. They're stored in an ephemeral volume specific to each container which other processes cannot access.

If someone has enough privileges to access /proc/*/environ of another process (i.e. root or the same user or process or child process) then they should be easily able to reach inside the container, no?

Other processes running as the same user can be blocked from reading /var/run/secrets via the likes of SELinux/AppArmor whereas they can't be blocked from reading /proc/environ.

It's a pretty fine distinction and I don't know how many people actually bother doing SELinux etc. in practice, but theoretically it's marginally better.

AppArmor can restrict /proc, see example from docker: https://github.com/moby/moby/blob/master/contrib/apparmor/te...

how does the application pick it up? We use the built in secrets that are injected into the container as env vars and then the application picks it up that way. Not trying to sound combative, just looking for better ways to do things.

Read the file on startup.

Bonus: you can watch the file for changes. Which means your app can pick up rotated secrets without a process restart, whereas if you inject secrets via the environment they're fixed for process lifetime.

it's both true

putting any form of secrets into a env var is deeply insecure

using env is the most simple uniformly available way to have configs

through I would take it a step further, anything which must not be leaked probably should not be in any config, weather env or file or similar, it should be injected through other means if viable

Over the years I've had a lot of discussions around 12 factor apps and encountered a lot of confusion. I find that the 12-factor site is great, but mainly for people that already understand why those things are important.

For people that don't already understand the why behind the rules, it took more in-depth explanation. For that purpose, I made the video: "What are 12 Factor Apps and Why Should You Care?"[1]. I know of a couple of companies that use that for new-hire orientation for engineers/devops that have said it's been really helpful.

Regardless where you get the information, it's worth taking an hour or two to learn about 12-factor apps. Most of these "rules" are just things that take awareness and aren't immediately obvious unless you've learned them all the hard way by making the mistakes and feeling the pain.

[1] https://youtu.be/REbM4BDeua0

It's like unit testing. The idea of writing code to test your code was so blindingly obvious to me from the time I started programming that I was shocked that it took so long for other people to pick up on it, but the concept of "unit testing" started to gain some traction around the late 90's and I was relieved that somebody with some authority was advocating it.

But the reason I didn't just do it myself before Kent Beck published JUnit was because the code I worked on didn't lend itself well to being controlled by other code. Global variables everywhere, widely distributed state, dependence on external systems and specific file system layouts and general disregard for modularity made it impossible to run anything outside of the context it was designed in. All of this was "bad design", but it met deadlines, so everybody did it anyway. I was _hoping_ that if unit testing gained some traction, programmers would get away from monolithic design.

After 25 years of observing unit tests for getters and setters and then one big unit test that creates an in-memory database because every function in the app needs a live database just to run and then gets commented out because it fails, I've lost any faith that unit testing will ever be more than a meaningless checkmark that everybody fills out because it's a "best practice" but nobody actually stops to think why.

I've seen similar as well. Aggressive scrum has also made this much worse by laying all the incentives at "ship now and let the poor sap later who has to work this figure it out. I got my 'story points' and shipped so I won't be under the metrics microscope." Having a general requirement of "each code change should have tests in order to pass code review" can definitely help, but it doesn't incentivize good tests, and IMHO the only thing worse than no tests are bad tests.

I'd like to believe that "everybody" and "nobody" in your story aren't as black-and-white.

Many people see the value of unit testing. And see how TDD actually shapes your code to be testable and (accidentally, well, not really) therefore also cleaner and better organized.

I've always said, and will continue to do so: good, maintainable code, is accidentally also very well testable code. And vice-versa. And it's not a real accident, if you think about it.

But it only takes one person who either doesn't care or doesn't understand to make testability impossible for everybody else. And that person is usually management's favorite because he ships "on time, under budget".

I've ran from such places. Luckily for every such management, there are many more managements that understand the correlation between short term and long term gains.

I have always disagreed most with the config advice. Config ends up getting defined by a lot of different parties and quite often by developers and its often best to ship the application with reasonable defaults and then allow overrides from environment specific files and then environment variables. This is the most flexible for most server side applications because you often know what a setup should be and its best source controlled but secrets need to be injected at run time. Configuration needs to have cascading levels to make life easiest in dev, test and production without enormous setup time.

> application with reasonable defaults

defaults for dev or defaults for prod :)

If you say dev, you will eventually break prod. And prod defaults may not make _any_ sense for dev.

> And prod defaults may not make _any_ sense for dev

They are actually rather dangerous even. You certainly don't want your dev (or test CI) to reset the database if you are accidentally connected to prod. Same for email-endpoints, CDN, a message-bus, etc.

I've had my share of near-heart-attacks where I send out 10k emails or thousands of payments on actual production services because of a spagetti config loaders with stuff that boils down to `if (getHostname() === "localhost") { include("dev.json") } else { include("prod.json") }`.

I'd prefer a hundred .envs and whatnots to avoid just one of these scares.

My approach is:

- defaults are sensible (safe and secure) defaults for prod

- prod secrets are only available in the prod environment

So if you accidentally end up trying to run with the prod settings in a nonproduction environment it might try and access the prod database server... but it won't get anywhere because it doesn't have those secrets.

> Configuration needs to have cascading levels to make life easiest in dev, test and production without enormous setup time.

The backdrop of when the 12factor app was written is that you can toggle modes like that with RAILS_ENV=test (an environment variable).

One way I think of the config section is “does your config strategy play well with containers?” Once you build an image you have a static disk state that doesn’t persist changes unless you make a new image. If your config is file based (only), then to toggle between test and production behavior you would have to build a whole new image.

By allowing config to change without the underlying disk, it helps to isolate changes. Did the app break because something in the deploy (image generation) broke? Or is it a bad config? If you decouple image generation from configuration changes it eliminates that question.

> Once you build an image you have a static disk state that doesn’t persist changes unless you make a new image. If your config is file based (only), then to toggle between test and production behavior you would have to build a whole new image.

You could mount a volume with the config file. The nice thing is that this lets you change the config at runtime (assuming the app watches for file changes). Env vars cannot be changed at runtime.

> Env vars cannot be changed at runtime

You can specify env vars any time you boot up an image. I see this as a benefit. I use containers for production and don’t want wild ssh yolo modifications. I want to have very clear immutable inputs.

> mount a volume with the config file

Sure, but where? And what if you need multiple files for multiple different things to configure? What’s to prevent someone from accidentally putting non config files into that volume or accidentally changing the permissions on the file or any other host of things that we just don’t have to reinvent if we use env vars.

So long as you are systemically protecting against someone spinning up a QA instance against some Prod resource then it’s probably alright to ship most config with the app. You don’t want to allow a (entirely predictable) human error to cause 100k auth denied retries against a prod job submission queue from QA for example. Much config isn’t about infra though, it’s often what kind of ResolverStrategy (made up but plausible sounding name) bean do I want to wire in each env for example.

You can, and probably should, have reasonable defaults.

Configuration should be version controlled, but separately from source code because it describes deployments, not images used for deployments.

Why not also express the defaults as config, or is that what you meant? I don't think that runs afoul of 12 factor.

For sure an influential engineering codex. Feels almost strange thinking about all the easy abstractions we have now for hosting, like Render or Vercel or that this was written in 2012, when web apps were much much more of a wild west in terms of accepted/shared practices.

The big thing missing from this document is justifications for the rules. It's almost all just rules. Are the rules good? Hard to say, and this document won't help you find out.

If you click on the rule it links to a page that is what you're asking for. For example: https://12factor.net/backing-services

Not all the rules are explained on. For example, a detailed reasoning why there should be only one app per repository is not provided. It is only stated that it is not the 12factor way.

The page you link to does not contain a justification for the rule.

I was kinda expecting this from the title to be some sort of commentary on two factor authentication, like an app that needs passport photo, face scan, drivers licence, sms text, google authenticator, email link, password, fingerprint, all together to log in just once.

Your average centralized crypto exchange.

Sounds like roughly what the average stock or crypto exchange requires before they let you trade haha.

Thank KYC/AML for that.

Money is not data, it is a complex social construct. Moving money requires interacting with that social construct, i.e. government regulations.

Yes, crypto was supposed to get around that. No, government didn't agree.

Well I'm glad they at least aren't collecting skin samples yet.

Back in the early days of Docker, I did a whole bunch of work to make WordPress behave as a Twelve-Factor App.

It traditionally hasn't behaved as one – which sort of makes sense, because WordPress grew-up in a world where long-lived servers with writable and persistent local disk storage was commonplace.

I'm sure things have moved on since those days. This was back in 2016. But it sure was a fun challenge!

I remember when I learned about making servers stateless by storing session information in a database and just not writing to disk and stuff. I was amazed at how much simpler it made things and added the capacity to load-balance multiple nodes without having to bother with session stickness.

Of course it made things harder in other ways like having a separate DB for sessions.

Previous discussion- https://news.ycombinator.com/item?id=31198956 (102 comments)

Another interesting thread about how these practices have held up or changed over the past decade:

"Twelve-factor app anno 2022"

Article: https://xenitab.github.io/blog/2022/02/23/12factor/

Discussion: https://news.ycombinator.com/item?id=31225921

The 12 factor app guidelines are still relevant today since many software teams are building distributed systems, which require careful attention to detail as to how services are coupled, configured, and deployed. The 12 factor app is a wonderful starting point for best practices. In my experience though they omit a few facets and leave room for antipatterns in certain facets.

https://smallbatches.fm/5/transcript

[00:03:52] Joe Kuttner: But today the infrastructure that we run on is disposable, right? It's much more like livestock or cattle where we can buy new ones when we need one at market and we can dispose of, you know, something's not working. Right. We get. Yeah. So it's a very, it's a very different model.

[00:04:07] Joe Kuttner: And that was just emerging when the 12 factor app was first launched. I think it did a great job of helping bring people into that state of mind.

https://smallbatches.fm/20/transcript

I love this generally but some of it has leaked out to the wrong people and there are strong cargo cults around such things as one codebase===one app.

Nowadays, I see many articles titled "Beyond 12 factor Apps" which proposes even more factors for App prominent among them being one from IBM. What's your opinion on that?

https://developer.ibm.com/articles/15-factor-applications/

This is a small but real trend. My experience at a company that went deep into expanding beyond/removing some of the 12 Factors is that it breaks portability of containerized software built with those assumptions. There were a few times I needed to install a vendor product or open source project that was built to the 12 factors but spent a month or more just building a shim to make it conform to our particular bend on it. I’m sure there are good reasons to deviate but I wasn’t particularly impressed with our decisions and more importantly why we made them.

I wish there was a generic control plane for generating and shipping config to apps. Kinda what Envoy proxy has, but capable of shipping arbitrary schema.

Many complications of config management will go away then. Every app instances knows just its ID, then everything else (region, cluster, stage, current experiment, maintenance, anything else) is discovered by "configd" service and config suitable for that very instance is returned. Not unlike feature flags, but for backend services.

OPA (Open Policy Agent) comes close: it has data sources (so it can fetch inventory from cloud providers, kubernetes, any other internal system) and Rego language to program config generation, but it doesn't have means to notify instances of config updates, so they have to resort to polling.

It's pretty bad that, more than a decade after this was released, there are detractors for some of the guidelines but yet nobody is able to point to a singular more modern set of principles. A lot of our advice is now nuanced and spread out all over the internet...

Seems good as a general heuristic but bad as a requirements checklist.

I genuinely feel like there's two camps of professionals: those who know how to do something and do it extremely well; and those who know how to describe something and break it down very well. The former are the ones making the magic happen, like Fabrice Bellard. The latter become very influential teachers like Bob Martin. Both are needed (although the former more so), but the latter usually get all the attention and credit.

You should add 2017 in the title.

yeah I got confused seeing this on the top. in my head said "what is this, 2012?". Am I wrong this existed way before 2017?

edit: I am correct, it is much more old. https://github.com/heroku/12factor/graphs/contributors

First commit was 2011.

The 12 factors go back even further, to Joel Spolsky's "Joel Test" in 2000:

https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...

That's a rather different set of 12 factors.

I hate things like these. They just make people architects and SEs create forceful indirection that creates nothing but burden.

Perhaps in 2023, but in 2011, we were still at the edge of when people would deploy by moving folders and individual files via FTP, hard-coding credentials, and dropping in extracted zip files as a method of dependency management.

Of course even then added layers were often considered "burden", but in general, we need some degree of order lest we create an unmanageable mess of index.old.3 and files edited directly on production.

Thought we are up to 15 now? https://domenicoluciani.com/2021/10/30/15-factor-app.html

It's good to see it coming up again. I hope there will be a platform among the new PAAS players that is 100% compatible with the 12-factor app principles.

Kubernetes basically forces you to deploy 12fa, using these guidelines is a pretty good playbook for migrating an existing application as well.

I prefer to have known configuration presets (groups) version controlled. Documents known shared environments (db host, ... etc) well.

Fun to compare this to with how insane deployments are now featuring kubernetes manifests which only big brains can understand

What part of the Kubernetes manifest do you think makes deployments insane?

Not GP, but here's an example I've seen. Kubernetes files are generated by Jsonnet. It is invoked from makefiles. Makefiles are spread across different git modules. The scripts require various Go binaries. Most are installed through a command, some have to be installed manually. If the version is wrong, mysterious errors happen.

I once had a problem with the generated files so I deleted the generated directory and regenerated everything. Turns out you can't do that. The generated files are edited, and if you generate from scratch you've wiped out the manual edits.

Note that I'm not criticizing Kubernetes here. But when you have a non-simple system with some missing aspects, some people take the opportunity to make it even more complex.

The superset of 12factor is platform engineering, especially managing stateful backend components.

I worked at a MAANG where both manual discovery (documentation) AND automated discoverability were completely haphazard and often missing because everything was home-grown.

Important takeaways should be:

- Don't reinvent everything without a specific problem that cannot be solved with reasonable scaling of what already exists.

- Theory of least surprise.

- Churn is bad. Don't change APIs or functionality for "refactoring" reasons. Replacing something with Rust is no excuse for leaving it broken for a month.

- Don't duplicate functionality without differentiated purposes.

- Make the purpose of a service and it's use-cases crystal clear.

- Name things sensibly rather than giving them cute names.

- Standardize stateful storage into a common API that allows for interchangeable and migratable backends.

- Document everything into a wiki that's consistent, organized, and is maintained by dedicated, effective technical writers. Don't brain spew text into massive documents!

- Following the previous, have a comprehensive service catalog.

- Don't assume the reader knows what you know. Explain everything from primitives.

- Aim for self-service.

- Service owners and/or their operations support must be professional and available enough to provide input when self-service, board messaging, and peer discussions are unable to provide an authoritative resolution.

- Reuse, reuse, reuse, and reuse some more.

- Build tools that can visualize, chart, and record query results from almost anywhere.

- Don't be Twitter: Build selective authorization into every internal API.

- Least (but necessary) privilege: issue enough reason-tagged authority delegations such that people can do their jobs. Locking down every bit of internal telemetry and analytics preventing joins between datasources doesn't scale OR create a situation where people are perennially in technical violation of the wording of official policy to do their jobs.

- Reward the reduction and efficient maintenance of code because too often, only adding features is rewarded.

- Don't make the common cases difficult and awkward.

- Avoid code generation.

- Avoid unnecessary layers of abstraction.

- Avoid building layer upon layer of CLI codegen or service commands lacking in documentation and missing man pages.

- Hire some people with actual software engineering, software architecting, systems engineering, and systems operations experience. (The names and responsibilities of these roles seem to change every 5 years now without any particular logic.)

every time I read Twelve-Factor App I'm like who they set up twelve factor authentication, that's crazy isn't 2FA enough

then I remember that it's about something else

Would be nice to have a simple term for all these absolutist ideas about how to solve a Problem. I nominate the term "The Holy Grail Jail".

I've actually had great success doing the opposite to most of these points.

"last updated 2017" if you look at the footer

Holds up well 12 years later.

Ask HN: Is 12factor.net Still Relevant? - https://news.ycombinator.com/item?id=36283702 - June 2023 (6 comments)

12 Factor App Revisited - https://news.ycombinator.com/item?id=33164407 - Oct 2022 (7 comments)

Twelve-factor app anno 2022 - https://news.ycombinator.com/item?id=31225921 - May 2022 (35 comments)

The Twelve-Factor App (2011) - https://news.ycombinator.com/item?id=31198956 - April 2022 (102 comments)

Twelve-factor app development on Google Cloud - https://news.ycombinator.com/item?id=21415488 - Nov 2019 (63 comments)

The Twelve-Factor App - https://news.ycombinator.com/item?id=19947507 - May 2019 (3 comments)

12 Factor CLI Apps - https://news.ycombinator.com/item?id=18172689 - Oct 2018 (247 comments)

12 factor app configuration vs. leaking environment variables (2014) - https://news.ycombinator.com/item?id=15869436 - Dec 2017 (2 comments)

Ask HN: Alternative to Heroku that doesn't enforce 12-factor - https://news.ycombinator.com/item?id=10628961 - Nov 2015 (1 comment)

The Twelve-Factor App - https://news.ycombinator.com/item?id=10288216 - Sept 2015 (3 comments)

The Twelve-Factor App - https://news.ycombinator.com/item?id=9492120 - May 2015 (2 comments)

Twelve-Factor Applications with Consul - https://news.ycombinator.com/item?id=7780249 - May 2014 (2 comments)

The Twelve-Factor App - https://news.ycombinator.com/item?id=7547687 - April 2014 (1 comment)

Building Twelve Factor Apps on Heroku - https://news.ycombinator.com/item?id=6219444 - Aug 2013 (1 comment)

12 Factor model for architecting SaaS applications - https://news.ycombinator.com/item?id=6060381 - July 2013 (1 comment)

The Twelve-Factor App - https://news.ycombinator.com/item?id=5979452 - July 2013 (1 comment)

12factor: Methodology for Building Software-as-a-Service Apps - https://news.ycombinator.com/item?id=4027026 - May 2012 (1 comment)

Twelve Factors of Web Application Development - https://news.ycombinator.com/item?id=3267187 - Nov 2011 (37 comments)

makefile cackles as it observes the exchange

Not this again. Let the horse die already

This horse or any other horse. I read this and I think "this is good advice, we should follow it." You read it and you think "this is good advice, we should follow it." And then we both do, but our interpretations conflict completely so we end up arguing about the true definition of "implicit existence of system-wide packages".

Sorry, i put all secrets in my code. I don't need 12 factors at all.

It's super easy to deploy to multiple environments, because code is single source of truth.

Secrets in code is bad practice because it puts the secrets in source control and gives them to anyone with access to the code.

You separate out the secrets to keep them secret.

Next evolution of Git should automatically introspect the code base for comment to leaving out all "secrets".

My point is, make the code easy to clone in a runnable shape wihtout any hacking on host system.

You can configure GitLab to reject the push if you comitted secrets, based on regex marching with common formats for tokens.

What company do you work for? I'd like to cancel my subscription.

Seriously...bad idea. One repo leak or even one bad copy/paste in StackOverflow and you're done.

It's not a problem. Here you go: https://github.com/AGWA/git-crypt

This only works if secrets are frequently rotated and contain enough entropy. Else, all secrets will eventually be decrypted from leaked repositories and distribution artifacts.

Hell yeah brother. Move fast and break things philosophy

Boring. This is already happening in any sane modern application.

It might be worth you reading https://news.ycombinator.com/newsguidelines.html

In particular:

   > Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.

   > Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something.

> This is already happening in any sane modern application.

Probably in part thanks to 12factor.net

Reply on news.ycombinator.com