How We Build Code at Netflix

As for writing a plugin, no, it would not have been possible. Concourse has an entirely different model of operation and needs easy access to containerisation facilities to achieve it.

Concourse doesn't really think in terms of "plugins". What you become accustomed to is wondering "is there a resource type for this?".

Right now I work on a software repo with a moderately complex API for uploading final binaries and releasing them to clients to download. Instead of telling people to write scripts or install a plugin, I can point them to the resource that another team has written.

"Just add the resource". Released software is now just a stream of events, no different from git commits, S3 files, points in time, Tracker stories etc etc. Every resource has the same interface so it makes it possible to click together stuff into clever combinations, rather than lashing together awkwardly and hoping it'll work.

My first approach was as an individual, which was quite difficult, because it took me a while to work out the differences between jobs and tasks and how to lay it all out. Lots of copying and pasting from other pipelines I studied.

At this point pretty much every team in Pivotal's Cloud Foundry division is running a Concourse pipeline, including the one I belong to. What's been interesting is how each team is experimenting with patterns that Concourse makes possible.

For example, I and my peers are now turning various things into "executable documents" -- the terminology sucks at the moment. Think of all the stuff buried in READMEs and wikis and long-forgotten cron jobs. How to build that special docker image that you only update every few months. The database backup. Keeping the blue-green deploy codepath warm.

When we find another one of these, we now encode it into our pipeline. That way, if I need to find it, it's there. And if it goes bad, the pipeline definition points me at where to go looking for everything of interest.

Another pattern that is emerging for us is "enforce project invariants". For example, we have multiple repos, so multiple Gemfiles and Dockerfiles, all of them with ruby versions set. At the front of our pipeline there is a little gateway to check that these are all identical. If not, it prints a table of versions found, so again, I know instantly where to go look. We have various other little invariants that turn days of insane debugging into a few seconds of sanity-checking.

None of this is novel. What Concourse lets me do is hoist good design practices out of code and apply them to CI. The final feedback loop I rely on can itself now be checked in, broken apart according to SOLID principles and even (my colleagues in Buildpacks are pioneering this) unit tested. Some colleagues in London have designed a fully generic pipeline that can be configured at runtime to build any of the five products they manage.

tl;dr fuck yeah Concourse.

I maintain a personal server that hosts a variety of VMs for things like mail, static site hosting, nameservice, etc. Last time I rebuilt it, I thought, hey, I'll try doing it with these interesting tools. The immutable-infrastructure approach turns out to be pretty heavy in that case. A lot of times it feels like I have to perform a triple bank shot just to make a minor configuration change.

That said, I think the one-off server, carefully maintained by one person, is becoming very rare. When I build things that other people will work on, I think immutable infrastructure and the cattle-not-pets approach is the only responsible way to go. Even if traffic volume won't be huge, I think the clarity and ease of debugging you get is vital when it doesn't all live in one wizard's head.

I agree that stateful servers are a challenge in this context. But they're a challenge regardless. Having that one thing we're afraid to upgrade or restart can cause all sorts of development and business process issues. That might have been a worthwhile tradeoff 10 years ago when you had to buy all your hardware and you needed ops people to do a lot of stuff manually. Servers go down, and we might as well accept that from the beginning.

For example: update the code in your source control repository, and then build the new version of the software.

    FROM ubuntu
    WORKDIR ${foo}   # WORKDIR /bar
    RUN developer_script.sh

If you just need to make sure your developers don't make a mistake in terms of securely configuring their containers (and making sure to always use the latest software) then you simply scan them before the canary stage. The problem there is, "what are you looking for?"

Also, there exists only one tool to scan Docker containers (OpenSCAP) and if your security team doesn't like it, well, you're screwed: https://github.com/OpenSCAP/container-compliance

The other problem is that OpenSCAP only checks the container's packages for compliance. It doesn't actually scan the container's filesystem for things like JREs and bundled libs. So if you're using Docker best practices by keeping your images as minimal as possible you may not even have a package tool inside your containers. In that case how do you check for things like out-of-date versions of Java?

Another problem is you assume there's some modicum of control over what's inside the containers before they're deployed. We're in charge of creating the infrastructure for running containers with the promise that end users (who would be various application teams) can make their own containers (or at least their own Dockerfiles) and deploy them on our infrastructure.

We can work with them to help develop, say, a Kubernetes pod config for their app but as far as what their app is or what gets bundled with it we'll have no knowledge after the first successful deployment.

Our Docker registry pulls down the latest images (that we use with our Dockerfiles) multiple times daily. So if the author of the Dockerfile just puts "FROM whatever:latest" they are guaranteed to have all the latest/patched software whenever they "docker build".

Our main issue is also security - as docker containers aren't very good at actually containing, from a security PoV, we're using SELinux MLS, assigning each container a unique SELinux level/category (fedora-atomic sets this up nicely for you if anyone wants to try).

This is great for security, but breaks so many things - lots of the cutting edge work with docker and related tech isn't done by people that care deeply about security, or really understand this sort of usecase, so we run into a huge amount of edgecases and bugs. We spend a lot of time fighting this.

It also makes having a really good build/deploy pipeline essential, as well as a good environment set up to reproduce and diagnose issues.

It's saying your engineers are so much more precious and able than the barbarians outside the gate, that they should write their own test harness/DOM selection library/insert commodity software genre here.