Hacker News Reader

It’s not off-base, just outside of your personal experience. This shows in your first comment about updating when you deploy: think about a large and not especially functional environment where there’s stuff written by vendors, contractors, acquisitions which have been folded into various areas, etc. They might not be deployed very frequently at all, and nobody in ops is sure which can handle a rebuild, whether the upstream images for all of those containers are still getting updates or whether upgrades will break something else, etc. They track Red Hat CVEs but not Alpine, etc. The corporate security scanner looks at the base OS but doesn’t know how to introspect containers – or it’s Red Hat’s and doesn’t know how to handle anything which doesn’t use rpm, etc. Performance is similar: containers add a layer of complexity which requires a lot of tools and practice to change – arguably for the better but many places weren’t doing so well even before the problem got harder.

I still think containers are the best answer to a ton of operational needs but it’s absolutely true that better tooling is needed for a bunch of problems, and this is repeating the classic hype cycle where it’s being billed as a cure-all when in fact it’s still going to require time, staffing, and a commitment to do the job well.

As an example, start with the easiest problem: say you want to prove that you’ve installed the latest OpenSSL patch. On traditional servers, this is a well solved problem. If you’re using Docker, your options are to buy a commercial offering with a support contract or, if your purchasing process is dysfunctional, build something around Clair, which has a bunch of usable but not great tools. If you’re the ops person looking at that, you’re probably thinking this just made your life worse even if there’s the promise that in some indefinite future it could get better. I’m hoping that the OSS community starts rounding out rough edges like that because it’s definitely an enterprise adoption barrier.

How does "It auto-updates every time you deploy" fit with "The difference between my local, staging, and production is tiny"? It sounds like you have little control over your dependencies, and the differences between your local, staging, and production environments are the newer versions of your dependencies which you haven't tested against.

I like the idea of being able to precisely control both my code and all of my dependencies, so that I know for certain that I'm deploying exactly the same overall system that I tested. Containers are much better for that than the old way, because you could never be certain that your OS and system software were exactly the same in production as they were local and staging. But to achieve that precision, you need to use precise version numbers, and you need to install your dependencies from a local repository to be really sure.

> How does "It auto-updates every time you deploy" fit with "The difference between my local, staging, and production is tiny"?

It builds once, and then that very container with those dependencies gets pushed through testing, staging, and production. No more changes happen after the build.

Ok, that definitely works, assuming you set your process up that way. In my mind, "deploy" starts with build and targets a single environment. Eg: deploy to staging, deploy to production. I've been working with too many clients these past few years who do it that way. One has a real TeamCity CI setup, but they rebuild for production too.

Way back in the day, I helped push my company towards a process where build artifacts were tied to specific commit ids (SVN, back then) so that everything that reached production could be traced back through QA and Development. So, basically the process you described. No containers back then, of course, and no VMs either. We had real servers in our server farm.

I thought the very basis of continuous integration/delivery was the same build moving through various stages of readiness and environments until it’s delivered to production, automatically and continuously. In short, if it’s called continuous delivery, that’s about the only way the process could be setup with containers and still be called CD. Or am I mistaken?

I think that by "It auto-updates every time you deploy." the OP meant - every time you build an image to deploy. But that exact same image goes through testing environments. (test -> staging -> prod)

Local may test with something that's a point release further, but in that case you'd find the issue when testing and pin the python to a specific 3.6.x until you resolve the problem and start rolling again.

There's also nothing preventing your from building images with precise control over versions of your dependencies. You can do it in the image in the same way you'd do it anywhere else. Specify your own repos and use lock files, or whatever your language allows.

You're right, I didn't make that clear.

What you're missing is that the beginning of the "deploy" process is building the image on your local (or on CI). That's when the update happens. Then you test it on staging, and if all is well you deploy to production.

If there's a problem it's easy to change your Dockerfile from "python:3" to "python:3.6.2" in order to go back to what you had. Or stick with "python:3.6" if you only want security patches. Or, if you want to miss out on those security patches in order to guarantee more stability, go with "python:3.6.2" and decide when to test and deploy an upgrade.

Are you referring to Canary Builds?

https://www.thoughtworks.com/radar/techniques/canary-builds

No, I am more talking about a standard process of applying security patches (and/or bugfix patches). I'm countering the claim in the article that using containers somehow makes software orgs more prone to "unmaintained" OS's.

I'm saying that has little to do with containers, and if anything containers make it a lot easier to get security updates.

I happened to get thrust into a large orgs ops management position 4 years ago and got to be on the bleeding edge of usable docker and kubernetes. I'm right with you on this.

I was lucky enough to get to slog through this stuff hard and have been building containerized development and operations systems as a huge amount of my work ever since. I get the author's position and I think it's likely the case for most people if they are asked to "Just use containers." It takes planning, knowledge, and a pretty multiclassed skill set to put together great container operations but even from day one when I realized I could isolate node (way before nvm) it was a godsend.

I barely run anything on my base system anymore. Everything I put together is now a cascading series of helm charts that easily deconstruct into bare metal deploy. The developers on my team are able to move fast with it because I stay ahead of it and make sure the tools are usable and documented before they are even thinking about them. I can take really obtuse customer integrations and quickly come up with solutions that don't create friction because of how fast I can break them and "infra as code" their way into our stack so no one has to deal with the fact that the API is garbage. I deploy things with health and liveness checks, I get reporting across the board of usage. Anything I want to flight to the world or internally is authenticated through our LDAP/Directory/GHA and I don't need a server troll to administer it.

I fully understand people not wanting to use them and just stick to what they know, but containers are amazing and I use them at micro to macro scale. Like you, my code has never been so up to date.

It's fun to write a glib article about how you don't like things that are happening. Great if you don't wanna learn them soup to nuts, but to dismiss their value so absolutely really misses a ton of opertunity. Even if it's many many pain in the ass weekends to get fluid with it.

> It's just so damn easy, especially if you use `FROM python:3` instead of `FROM python:3.6.2`. It auto-updates every time you deploy.

There's an issue with that. You're trusting whoever builds the python:3 image to actually update it and be secure.

There are a couple high CVEs in python:3 image, including a 10:

https://security-tracker.debian.org/tracker/CVE-2017-17458

https://security-tracker.debian.org/tracker/CVE-2017-17499

Then there are a bunch of other medium and low CVE, mostly from imagemagick, which is kind of a shame to include if you really don't need it. Same goes for that 10 for mercurial if that's useless to your project too.

You are best off receiving a base image from a trusted source, eg, if your organization maintains a set, or there is some distribution you trust who provides just the OS. Grab the most minimal set, then add your application on top of that. Make sure you go through a check to ensure you're not adding any insecurities yourself.