12 Fractured Apps

I created this logrotate rule to rotate the docker json log files:

  /var/lib/docker/containers/*/*-json.log {
      copytruncate
      rotate 2
      size 40M
      missingok
      nocreate
      nocompress
  }

But for most services I still mount a log dir and redirect stdout and stderr to a file manually, since neither the json log format nor that directory structure are particularly convenient for my purposes.

More recent docker releases have added other logging drivers, and added options to the "json-file" logging driver for "max-size" and "max-file", which sounds appropriate, but that came after I worked around the problem and I haven't upgraded yet.

>The article starts off by talking about trying to run legacy applications and then just casually tells you to modify the source code to solve all your problems. Not everything you want to run in a container is something which you want to modify.

I think it goes without saying that practicality/pragmatism trumps convention or philosophy every time. I suppose the question we have to ask ourselves every day is "are we being resistant to change or are we being pragmatic?" Anyway, he does cover this when he mentions applications you don't have control over. So I'm not sure it's a fair criticism. This post seems to be aimed directly at developers or people with the latitude to effect change with them.

Using environment variables for everything is wrong too. API keys and other sensitive information should be in environment vars. Non-private information should definitely be in config files.

If you need the flexibility of environment variables for a semi-configurable non-secure variable, use them to overwrite a sensible default.

Environment variables are just a mechanism, or transport, for getting information from the environment to your process. You don't version or manage them, any more than you'd version or manage command line flags. Change control is the responsibility of whatever system sets the variables, or execs the template that produces the run script, or whatever. You don't actually set env vars on the host.

You effectively build your configuration file into the thing that knows how to run your container. If you're running Kubernetes, this is either a secret or the replication controller definition file. For docker-compose, this is the `docker-compose.yml` file. Or it's the script that starts your container.

But it's pretty common to put service credentials into a config file, so it's an anti-pattern to version-control them. It's _way_ safer not to, which means you shouldn't be version-controlling the thing that runs your container? This is sort of tricky. We're doing it by volume-mapping a non-version-controlled file for database credentials, and storing the rest of the configuration in the database.

> Two things which I never understood about using environment variables are how do you version control the changes and how do you manage these variables when you have more than just a handful of them?

If you're using Cloud Foundry, you put them in your manifest.yml and check that into source control. When you do `cf push`, they'll be updated.

Disclaimer: I work for Pivotal, who donate the most engineering to CF.

> Two things which I never understood about using environment variables are how do you version control the changes and how do you manage these variables when you have more than just a handful of them?

We're doing this: the env vars are stored as a stage/container/key hierarchy in version-controlled eyaml files (yaml with encryption at the value level, nice for git diffs). At deployment the eyaml gets decrypted by ops or jenkins converted into a container env map (in our case a kubernetes resource controller).

Additionally we tag deployed containers with the config's git hash to have reproducible deployments, which is actually pretty useful. (again we leverage kubernetes labels, but this principle should could be applied to other orchestration tech i guess).

You're welcome.

You have managed to capture the spirit of the post. The goal was to highlight areas where developers can take action, and how improving even the little things can go a long way to improving the entire system -- even the one you can't see.

In the early parts of my career I would often take pride in building complex systems to accommodate for misbehaving applications. Throw in some fancy Nagios alerts and a sleep depriving on-call rotation; I looked like a hero.

Then I learned how to write code.

This was the turning point in my career (2006). I was now brave enough to modify "legacy" production applications to take advantage of "new" infrastructure features like service discovery (use DNS records instead of IPs), and logging directly to syslog (asynchronously with proper ring buffers).

I was willing to learn any language too: Python, PHP, or Java, it did not matter because it allowed me to take action and contribute at the heart of the application.

I'm not saying platforms that also handle "misbehaving" applications or complex failure scenarios are unnecessary. I just consider those platforms as extra layers of protection, not a free pass to ignore building applications that take responsibility for reliability from startup to shutdown.

This is a problem that several new tools are being built to address, e.g. Vault from HashiCorp and Keywhiz from Square. Storing the secrets unencrypted on disk on the host system is not a huge improvement over having them in the application by default. Ideally you want a system to store them securely that allows them to be extracted and decrypted only using credentials and policies you control. They should only ever exist in memory (which is why Keywhiz uses FUSE, for example.) Some container orchestration tools like Kubernetes also include their own mechanism for securely storing and retrieving secrets and making them available to applications.

One way to think about this that maybe overlaps with the theme of this article, but also stands on its own with specific regard to database modifications is that you often need to have the ability to have multiple versions of everything alive at one point in time.

So maybe that's a schema A and a schema B, or maybe you have applied schema B which only app version 1.1 is optimized for , but version 1.0 is what is in production immediately following your database migration. So you can't make changes in schema B that would immediately render app 1.0 broken, which means you need to not box yourself into a corner with future assumptions as much as possible.

Ultimately if downtime is not an option you end up writing these capabilities in at every layer. Whether it's an api endpoint or code talking to a database you often have to make carefully thought out changes incrementally to ensure that things can all operate simultaneously, and often this ends up having metadata about the versions of everything as an option for taking different code paths.

This article touches on this in the way that if suggests making your app deal with an available database and one that is not available. Same with a field in a schema or a payload. To make your code less brittle instruct it what to do in both cases.

Threat your database like an API and only introduce backward-compatible changes. For example if you want to rename a column, make a new column with the target name and a trigger that keeps both columns in sync. Once the new code is deployed everywhere you can remove the trigger and the old column.

I think that author just does not have experience outside of what he does. Maybe his systems can fall back to sane defaults. But what is sane default if you have to communicate with 3rd party server and your system is worthless when connection is not there? You have to have ip/domain name configured.

For cars if something is wrong then in some cases you can start and even drive but users get warning. If there is something really wrong car will not start.

So I think what author suggests is at least asking for trouble.

Almost everywhere as quoted:

"Everything in this post is about improving the deployment process for your applications, specifically those running in a Docker container, but these ideas should apply almost anywhere."

I agree. Things become incredibly murky if preconditions are not clearly seperated from optional settings with sensible defaults. For a server-side multi-user application to just go off hunting for data stores or even create new ones whenever configuration settings are missing seems like a security and data integrity nightmare.

If something is a precondition then the app shouldn't act like it wasn't just to make Docker configuration easier. It needs to fail fast.

Retrying database and other connections is sometimes the right thing to do in long running applications. But I dont' think application launch is the right time for it. Application launch is an opportunity to make sure that all dependencies were in place at some point. If things break later on, the odds of it being a temporary issue are much better.

"Providing the location of the backing services" doesn't mean the physical address, but the logical address, managed by an e.g. load balancer, capable of abstracting over physical changes as necessary. Consul is one way to do it; DNS is another, and there are many more.

CNAMEs are perfect for this. "customer-data-service.my-product.com" could point to absolutely anything. Give your CNAME a name which is meaningful no matter what service is behind it and you'll never need to change it (externally).

any problem can be solved with a layer of abstraction.

> Do they expect you to restart when you switch backing services? Do they expect you to run all your internal traffic through load balancers?

12 Factor was pioneered by Heroku, so yes.

Edit: I don't understand the downvote. This is the factual answer. 12 Factor apps are supposed to delegate a lot of cross-cutting concerns to the platform, including the problem of "where does that service live?" and "can you give me another chunk of service?"

I agree with the general idea, let an already existing product handle problems that everyone has. The last part is still important though, if we are going to build "distributed" apps, then we need to handle those dependencies failing or being briefly unavailable.

And the reality is that every app becomes distributed as soon as it has a database or client. A ton of legacy applications make the implicit assumption that the network is reliable, and fall over hard when it isn't true anymore.