Hacker News Reader

Rather than using Puppet etc.? Basically you've returned to before configuration management and that is somehow good?

How do you change network configuration, load balancers or other external configuration in lock step with your application? Scripting by hand?

And deployments are a just small part of configuration management. How do you find out which of your applications use libz 1.2.3 (to measure CVE applicability for example)? By looking in each and every one of them?

How do you find out which application run a cron job? And which application connect to backend x without going through the load balancer?

Which application has a client certificate about to expire? How do you guarantee two applications have the same shared secret, and change it in lock step?

With configuration management all this is just a look up away. And best of all, most of this information is authoritative.

When I come somewhere new, the use of these tools instead of a directory of miscellaneous scripts is like night and day. It literally turns a two day job into a ten minute one when the environments are complex.

Docker is great for a lot of things. It enables to do daring things to your environment even if it's so complex you don't fully understand it all because you can just shuffle containers around. But I would never use it instead of configuration management.

Re: '...Rather than do all this work with Puppet, Chef, or even Ansible, I can just declare what the systems ought to be and cluster them within code branches...'

It sounds like the poster is doing something similar to a 'Gold System Image' though with Container technologies.

Configuration Management is great, but I'm with the view that you should start with a combination of gold images/system builds and then stack on Cfg Mgmt on top of that.

Theory is you will have a known base and identical builds etc, otherwise you would get subtle drift in your configurations, especially for things which are not explicitly under control of Config Mgmt (shared library versions, packages).

Of course, this is not always feasible but if I were to start from scratch, I would probably try to do it this way.

It should be a combination, really.

You want to have your "Gold system image" available and this is most certainly what you should be deploying from; however your configuration management - whether it's Chef, Puppet, whatever - should be able to take a base operating system and get it setup completely from scratch.

This then solves the problem of ensure that you can completely reproduce your application from scratch; but also removes the possibly horrendous slow scale up time by using a "Gold image" that has already done this.

My current process is: Jenkins runs puppet/chef, verifies that the application is healthy after the run and everything is happy, calls AWS and images the machine, then it iterates over all instances in my load balancer 33% at a time and replaces them with the new image resulting in zero-downtime. Of course another solution is to pull out those instances and apply the update, and then put them back in.

And I'm sure someone else will have their $0.02 on their own process, which actually I'd love to hear :-)

Here's mine, with the preface that we're still iterating our deployment procedure as things are quite early.

I have an Ansible repo with an init task, which configures new boxen (makes sure proper files, dependencies etc. exist). Then to deploy, I have another task that ships a Dockerfile to the target boxen group (dev, staging, or prod) and has them build the new image, then restart. This happens more-or-less in lockstep across the whole group, and scaling up is relatively easy - just provision more boxen from AWS and add the IPs to the Ansible inventory file. Config is loaded from a secrets server, each deploy uses a unique lease token that's good for 5 minutes and exactly one use.

I'd love to hear how to improve this process, since I'm dev before ops. My next TODO is to move Docker image building locally and deploy the resulting tarball instead (though that complicates the interaction with the secrets server).

> Basically you've returned to before configuration management and that is somehow good?

I would argue I accomplish most configuration management via the Dockerfile. In Docker 1.7, even if I want to use a different filesystem like ZFS, I can.

I agree with you that I probably have missed many important features in my architecture. However, it's simple and works for what I got. It sounds like in the future, I'll need a combination of Puppet / Chef & Docker rather than relying purely on ECS. For now though, it's quick & easy. Thanks for the thoughts, they all make me think.

What happens if there is a security flaw somewhere in your software stack? (kernel, web server, etc.)

Do you end up having to update multiple containers with the same patch?

Context: I've looked at Docker, but not used any containerization yet.

Security ? How do you find out what libraries / versions are in a container ? ... you are pulling the curtain dont look back there :)

> Security ? How do you find out what libraries / versions are in a container ? ...

The same way you do in any application, container or not - you look up your documentation ;)

Is that really the state of things? It is what I concluded after looking into this a bit for Docker, but it seems incredible to me that so many companies are jumping into this idea of containerization without any good & available solutions for this problem.

One potential solution that came to mind was that if there was a standard way of deploying an application into containers, and Google/Amazon/Microsoft provided auto-updating containers, the maintenance of a secure container would be in the hands of companies who (hopefully) have the resources necessary to keep the entire stack up-to-date.

They seem to be betting the farm on the containerization will contain (heh) whatever security issues that come up.

This in the sense, i guess, that if they have a security flaw in their php that gives disk access, all the attacker will see is the content of the php container as the database will be on the next container over.

Then again the containerization seems to have come alongside devops, where the mantra seems to be "update early, update often, to hell with stable branches".

I've heard of that approach (breaches being limited to a container), but I don't think it makes sense.

If a security flaw exists in one container due to the stack not being updated, isn't there a pretty good chance that it also exists in the other containers?

Also, for any given container, there probably still is a way for an attacker to do immense amounts of damage. With the database container you can steal customer data. With the PHP container you can remotely instruct the database to do whatever you want, or just point the code at your own database.

Depends what the security flaw is. If it's (as discussed above) a php error, it's unlikely to be in your database container.

We tend to handle that part inhouse. We're using Jenkins and have it set up to build a standard set of base images with all the latest updates daily. It can be run on-demand as well.

All containers running code are based on these images, so the updates are picked up on the next build/deploy.

If you are SSHing into a container, interrogating the version, and upgrading it manually, you're doing it wrong.

Pull a new base image, have your build system rebuild and deploy the relevant containers.

What do people think about building static binaries in Docker containers? I'm a big fan of the busybox base image and nothing else.

When building a container, I personally look at the docker-alpine [1] image and see if I can just use that. Since it's musl instead of glibc, there are some incompatibilities with things like Oracle JVM, but it's a great runtime container for dynamic languages like Ruby and Python, especially if you need some additional libraries or helper programs.

Love it for quick scripting containers (like netcat) too since it downloads in line 2 seconds.

[1]: https://github.com/gliderlabs/docker-alpine

We've had some success with auditing containers and saving the results to a DB (for instance, running containers you can easily exec, say, 'dpkg -l'). If a package has a security issue, you know which containers have it, and you kick of your CI/CD to rebuild and redeploy with the fixed package.

I don't see how containerization changes your security strategy in this regard, except instead of ssh'ing into a system and running `apt-get update && apt-get install ...`, you `docker build && docker run`

I'd love to hear how you're handling the ECS part of this. From the experimentation I've done, it's not just a matter of telling ECS to update the service unless you're running 2x the number of container instances you need -- it won't pull resources out from the existing service revision in order to spin up the new ones.

Certainly can be worked around, but it's a little annoying.

There's two ways to go about this:

1.) Create a new task and update the service with the task. From the documentation (http://docs.aws.amazon.com/AmazonECS/latest/developerguide/u...):

> If you have updated the Docker image of your application, you can create a new task definition with that image and deploy it to your service, one task at a time. The service scheduler creates a task with the new task definition (provided there is an available container instance to place it on), and after it reaches the RUNNING state, a task that is using the old task definition is drained and stopped. This process continues until all of the desired tasks in your service are using the new task definition.

2.) (way I'm doing it) Update the desired count to 0 and utilize wait_until (http://docs.aws.amazon.com/sdkforruby/api/Aws/ECS/Client.htm...) services are stable. After this, update desired count to 1.

You're going to get some downtime with #2. I'm not sure about #1 because I don't understand how something like an API service could be run without error'ing out that the port it's trying to bind to and forward is not available. So I'm not sure how it would ever reach the running state.. but I haven't experimented going about it that way. I can handle the small amount of downtime.

Thanks! Yeah, those were the ony options I was aware of, but the "provided there is an available container instance to place it on" issue is a problem due to port availability, and I was looking for zero-downtime deploy options.

Re services sharing, we are actually hacking on a database of useful containerized services which are hosted on Github and can be run anywhere through HTTP or JsonRPC. Product is super early, but would love your feedback on what you'd like to see: www.stackhut.com . Just about to push ability to submit your own.

I think the key thing is that users shouldn't have to have a knowledge of Sysops or Devops to be able to take advantage of the benefits of containers. Most developer we've spoken to don't have time to learn a completely new paradigm, they just want the benefits.

ECS is pretty exciting, but isn't the same level of abstraction accomplished with the GCE/Fleet/k8s? Even the Kubernetes or Fleet install process on EC2 lets you quickly forget about the actual boxes.

CaaS is definitely the future of cloud. However I don't think ECS (IaaS+Container) is the way to go. In this approach, you still need to take care of the cluster capacity planning, scaling, failover, etc. And at most time, there always are some capacity sitting idle in your ECS cluster.

The evolution of CaaS would be sth. like hyper.sh. Due to its hypervisor nature, the whole cluster+scheduling thing can be made transparent to developers, who simply "compose" the app spec and submit to run.

I am a more front end heavy web developer. I know my way around apis and what not, but what you are describing sounds like backend bliss. I don't even know where to start but is there someway you can share more about the magic of your process? Like an example of a github repo of search and how you plug that into an app?

Either way, thanks.

Would love to see this as well. I know my way around the CLI but not enough to even grok what Docker does. Any good overviews/getting starteds/tutorials out there?

This is a somewhat novel workflow from what I can gather, so no. If you worked backwards from my flow though, you could figure it out no problem. Maybe I'll do a blog post in the future about it. I just don't have the time these days.

Hey, we're actually hacking on solving this exact problem at StackHut.com, which is a platform for containerised services that you can call immediately through HTTP. I think your sentiment is super-common: building microservices / container-enabled services and getting all the good stuff that comes with them is still too much of a learning curve for most devs. We're just about to push live the ability to add new services: would love to get your thoughts on what you'd like to see: leo@stackhut.com

I don't understand how you configure your host-specific servicc dependencies, e.g. connection strings. Ansible (Puppet, Chef) does variable interpolation / templating for this purpose.

I have configuration files for each environment, so `dev.config`, `staging.config`, etc...

Do you deploy the same docker image to each environment, and somehow retrieve the configuration files on startup? Or do you rebuild your images as you promote from dev to staging to prod?

Yes same docker image, but with different environment variables passed in for dev / staging / prod.

This is a hard problem, but outside the scope of Docker. There are numerous cluster orchestration/management systems springing up around the ecosystem. (We don't necessarily have on ready for prime-time yet, but a lot of very good work is taking place.)

On a side note, for microservices that expose a RESTful API I would also take a look at Kong (https://github.com/Mashape/kong) for delegating common functionality like security or rate-limiting.

Made (and open-sourced) by the smart guys at Mashape.com. Thanks for doing this.