Google Embraces Docker, the Next Big Thing in Cloud Computing | Hacker News Reader

Google Embraces Docker, the Next Big Thing in Cloud Computing

wired.com

218 pointsspahl69 comments

Google has plenty of experience with containers already, since they heavily use cgroups and the concept of containers in their production environment for isolation and resource control, never mind the fact that two of their engineers had written much of the initial cgroups code. I talked about this in my "Introduction to Containers on Linux using LXC" screencast [1]. Briefly, Google is in the process of open sourcing their internal container code [2], there was a Wired article that talked about their container orchestration system [3], and finally, there was John Wilkes (Google Cluster Management), who talks about the container management system [4].

Docker adds very interesting filesystem ideas and software for the management of container images. Personally, I think we are on the cusp of a transition from VPS (xen/hvm) to VPS (containers). I also hope that Google throws some of their concepts at the Docker project. Interesting times for this space.

[1] http://sysadmincasts.com/episodes/24-introduction-to-contain...

[2] https://github.com/google/lmctfy

[3] http://www.wired.com/2013/03/google-borg-twitter-mesos/all/

[4] http://www.youtube.com/watch?v=0ZFMlO98Jkc

Great points. Btw, Victor and Rohit from the Google LMCTFY team are now active maintainers of libcontainer. https://github.com/dotcloud/docker/blob/master/pkg/libcontai...

Also check out Joe Beda's deck from GlueCon: http://slides.eightypercent.net/GlueCon%202014%20-%20Contain...

Docker is a natural fit for GCE.

Great links -- thanks!

> Personally, I think we are on the cusp of a transition from VPS (xen/hvm) to VPS (containers).

I'm not so sure of that. I think a lot of the use-cases for VMs are based on isolation between users and making sure everybody gets a fair slice. Something like docker would work well with a single tenant but for multi-tenant usage docker would give you all the headaches of a shared host and very little of the benefits of a VM. For those use cases you're probably going to see multiple docker instances for a single tenant riding on top of a VM.

The likes of Heroku, AWS, Google etc will likely use docker or something very much like it as a basic unit to talk to their customers, but underneath it they'll (hopefully) be walling off tenants with VMs first. VMs don't have to play friendly with each other, docker containers likely will have to behave nicely if they're not to monopolize the underlying machine.

> I think a lot of the use-cases for VMs are based on isolation between users and making sure everybody gets a fair slice.

Containers do this.

Hm. We'll see about that. I can see a whole pile of potential issues here with 'breaking out of the docker' on par with escaping from the sandbox and breaking the chroot jail, which I see this as a luxury version of.

Of course you could try to escalate from a VM to the host (see cloudburst) but that's a rarity.

Docker seems to be less well protected against that sort of thing, but I'm nowhere near qualified to make that evaluation so I'll stick to 'seems' for now. It looks like the jump is a smaller one than from a VM.

Fair usage of resources and security isolation are two VERY different problems. Containers can be VERY good at resource isolation. Security has not really been figured out yet.

This isn't really a "we'll see" issue. It is a fact that containers do resource isolation. :P The security issues are orthogonal.

I beg to differ. If you manage to break out of a container then all the resources of the machine are at your disposal.

So they're orthogonal only as long as the security assumptions hold.

Containers don't isolate very well. One thing that is easy to do is to make the system do disk output on your behalf just by making lots of dirty pages, or make the system use lots of memory on your behalf due to network activity. And of course there are the usual problems that you already have with VMs such as poor cache occupancy.

Shared hosting of random antagonistic processes is something that many developers are not quite ready to embrace. If you are willing to run your service with poor isolation and questionable security then containers are just the thing. You'll definitely spend less money if you can serve in such an environment.

I don't know where this myth came from that you NEED VMs for fair slicing. The Linux (and most other OS kernels) have been doing fair slicing just fine for years. I think the disadvantage of containerization is similar to those of OpenVZ VPSes: you can't partition your harddisk and you can't add swap space.

It's not a myth. A VM is effectively a slice of your computer that you can pre-allocate in such a way that that VM can not exceed its boundaries (in theory of course, this works perfectly, in practice not always).

So all other things being equal, if you slice up your machine into 5 equally apportioned segments and you run a user process in one of those 5 slices that tries to hog the whole machine it will only manage to create 1/5th of the load that it would be able to create if it were running directly on the guest OS.

So yes, linux does 'fair slicing' if you can live with the fact that a single process will determine what is fair and what is not. That that process gets pre-empted and that other processes get to run as well does not mean the machine is not now 100% loaded.

Using quota for disk space, 'nice', per-process limits for memory, chroot jails for isolation and so on you can achieve much the same effect but a VM is so much easier to allocate. It does have significant overhead and of course it has (what doesn't) it's own set of issues but resource allocation is actually one of the stronger points of VMs.

Well yes, but kvm is a vm thats just using Linux to do this vm scheduling. The main issue is that the API for containers is less well defined (IO scheduling is not necessarily fully fair with VMs, but its mainly aio on the host side at least).

You can add swap space, and there is even swap space acounting support in the kernel. Personally I don't use swap, I just buy fat amounts of RAM and allocate them to diskless worker-nodes in my clusters. As for partitioning, manual partitioning can give a slight speed advantage (if you know which filesystem you want to use, you have a long enough lived job to justify optimization, etc.), but generally you can just use http://zfsonlinux.org/ or at least LVM2 to avoid the segregation requirement entirely. In the former (ZFS) case you get arbitrary-depth COW snapshots, dynamic reallocation, transparent compression, and other types of useful options for ~free, as well. In the latter (LVM2 LV) case you get single-depth snapshots (though in theory this is improving; eg. via thin provisioning) but no dynamic resizing support (AFAIK, unless you use nonstandard filesystems).

I want option 3. A 4U rack with 32 completely isolated embedded stand alone quad core ARM or PPC systems, a network switch and an FPGA on each connected to the switch fabric.

Then we can start doing some interesting stuff past finding new ways to chop computers up.

Very interesting, that would be something I'd buy just to mess around with, I can think of a few ways in which I'd use it right off the bat and if you give me couple of hours more I'll have a whole raft of them :)

32 ARM chips in 4U seems very low to me, just in terms of the TDP a 4U rack is able to dissipate at present. You could increase density a lot.

You could but I want standard storage per node (PCI-E FLASH), redundant PSUs and the TDP of a hefty FPGA going flat out is a lot larger than that of the ARM core.

That does not sound very high density compared to what you can get from a company like Baserock - http://www.baserock.com/servers

I want a hefty FPGA attached to the CPU bus and switch backplane. That will take a lot more power than the ARM core.

There's also the recent article about automatic and machine-learning based power and core management.[1], [2]

If anyone here specializes in similar things, I would be curious to know if this Pegasus system runs on top of or underneath Borg/Omega (or perhaps replaced it?), or is a separate system altogether.

[1] http://www.theregister.co.uk/2014/06/05/google_pegasus_syste...

Edit: [2] http://gigaom.com/2014/05/28/google-is-harnessing-machine-le...

I'm always surprised that OpenVZ[1] doesn't come up more in discussions about containers. I used to use it extensively in for small lightweight VPS's (usually just an apache or mysql instance) and always found it to be pretty impressive. I've used Docker to make a debian-packaged app available on a bunch of CentOS machines and it saved me a huge headache (the dependency tree for the app was huge) so I'm a fan - but still a little puzzled at OpenVZ's seeming invisibility.

[1] http://openvz.org/Main_Page

OpenVZ was basically the prototype for LXC. Distros seem to have better support for LXC since it's "official".

Yeah, I realise it's place in LXC history. It still seems slightly odd that it's been kind of overlooked. It offered quite a lot (and still does) that I don't see replicated in any of the other container packages. At least not without quite a lot of manual faff.

Possibly it was just a little ahead of it's time and was also overshadowed by the rise of HW virtualisation in the later 2000's. Having to install a custom kernel (certainly when I used it) was also a bit of a hassle mind you. Anyway - maybe someone will re-invent the toolchain using Swift or Node and it'll become cool again ;-)

It also has a security focus, while docker has started with convenience for deployment rather than looking like an isolated machine.

Don't forget Linux vServers as well.

It's probably because of the way OpenVZ is marketed (or should I say, not marketed). OpenVZ's technology could probably do the same as what Docker does but they're not marketing it in the same way. The concept matters just as much as the actual technology.

I guess. Having a commercial version with a different name to push can't have helped the branding either.

> Personally, I think we are on the cusp of a transition from VPS (xen/hvm) to VPS (containers).

There may be some of that, but I think more common will be continuing to have tradition IaaS bring-your-OS-image services, with a new tier in between IaaS and PaaS (call it CaaS -- Container host as a service), plus a common use of IaaS being to deploy an OS that works largely as a container host (something like CoreOS).

"Personally, I think we are on the cusp of a transition from VPS (xen/hvm) to VPS (containers)."

A transition back to, I think ... the very first VPS provider (JohnCompanies, 2001)[1] was based entirely on FreeBSD jail and was entirely container based, even though those containers were presented as standalone FreeBSD systems.

[1] Yes, verio did have that odd VPS-like enterprise service earlier than that, but JC did the first VPS as we came to know it.

Pardon my ignorance but don't containers share the same kernel as the host? Meaning I can't run a Ubuntu container in a BSD jail or vice-versa? I don't want to use containers if it limits my OS choice to being from the same family as the host.

For at least the last 15 years I have almost always done 'devops' myself (before it was called that). Managing deployment goes with designing and building systems.

My problem is that I am sort of stuck in the past. Whether I am using VPSs, AWS, or rented physical servers, I have only a partially automated way to set up servers. This scales to small numbers of servers just fine, and that is mostly the world I live in, but I need to improve my workflow. This really hit home yesterday when I had to upgrade a Haskell GHC/platform because I tweaked a Haskell app making incompatible with an old GHC 7.4.* setup on an older server, and ended up wasting some time before fixing things.

Working as a contractor at Google last year was an eye opener. I really loved their infrastructure. Eye opening experience.

Docker seems like my best path forward.

You have hit the nail on the head. The thing people want in the real world is improved workflow.

In the real world, everyone wants infrastructure to have the same sexy qualities: automated deployment (CD/CI), automated scaling, automated failover/high availability, automated service discovery (read: functional service topology dependency resolution), security, resource and capacity planning support, real time least-cost-overhead provider selection for third party infrastructure providers meeting some disparate set of performance requirements, etc. Unfortunately, it's not an easy problem area to deliver a one size fits all solution to.

Docker doesn't really have most of that stuff in scope yet, even vaguely. Actually, it seems to have a really weird scope: it wants to wrap different LXC implementations and other container-style unix environments (potentially supporting non-Linux platforms) but doesn't want to deal with managing the host systems themselves, having - kind of, for practical reasons (though not entirely!) - outsourced this to CoreOS (ie. some particularly specific configuration of a Linux host system).

Whether all of this recent Redhat/Google docker bandwagon jumping will amount to any real solution remains to be seen .. Google AFAIK effectively runs its services on fat clusters made of commodity hardware, organized in to segments ('cells'), running highly customised Linux distributions, and so does Redhat where HA is required. I'm pretty familiar with these configurations as I do this myself. So will we ever see meaningful support for other OSs? Other distros? Physical systems via PXE to support these clusters? Hypervisor guests managed with the same developer and operations workflow?

My wager is not soon, at least in a manner that everyone agrees on... Google will keep doing its thing (using its unlimited supply of internal, world-class nerds to deliver and manage services on their custom OS in a custom way because saving 1/2c a month per machine pays ten world class nerd salaries at their scale), Redhat will keep doing its thing (selling prebuilt systems at expensive prices that still comfortably undercut the likes of IBM, pretending they are manageable, but actually rejigging the whole infrastructure every system release leaving little in the way of realistic upgrade paths without expensive consulting) and you and I will be left wondering where that magical docker solution went that everyone was talking about in early 2014.

Google values devops / solutions, so I think we'll see some interesting announcements over the next 12 months.

Here are a couple of notes..

Deployment manager - https://developers.google.com/deployment-manager/

Saltstack integration - https://www.youtube.com/watch?v=0dOXbhenFl0

I agree, but that's like saying "I think the world will continue turning". Obviously, yes!

The deployment manager sounded interesting but I'm not seeing any support for arbitrary platforms (in the OS sense), or infrastructure providers (in the 'run it on my own hardware, or someone else's' sense), nor the opsier side (like business concerns separate to technology) of the ops part.

Some thoughts roughly summarised at http://stani.sh/walter/pfcts/

Thanks for sharing your thoughts. Great points and insights.

Following up a bit.. Google just announced Kubernetes[1], an open source container manager. Also, Eric Brewer is now on Docker's Governance Committee[2] to help push for open container standards.

Seems like a good step forward.

[1] https://github.com/GoogleCloudPlatform/kubernetes [2] http://googlecloudplatform.blogspot.com/2014/06/an-update-on...

I agree.. I had a very similar 'eye opening' experience as you did with Google's infrastructure. I am now working on a internal projects as a contractor / partner. Solid.

> My problem is that I am sort of stuck in the past. Whether I am using VPSs, AWS, or rented physical servers

Makes me feel old... I used to "deploy" with cPanel or DirectAdmin.

Is Docker one of those things that works better for big projects than small ones? I tried using it for 3 little projects, just to see what all the fuss was about. But every time it ended up being easier to use bare LXC.

Curious why it was easier to use bare LXC?

I felt like everything had to be set up outside the container, before starting it up. To get something simple like ZNC running, I had to write the config file beforehand and then add it to the dockerfile. I guess I can see how that would be useful in a big deployment. But for a one-off like this, it's much easier to boot the container, install ZNC, then edit the config. And if (when) I screwed up the config, I could just edit the file instead of rebuilding the container.

OK, that makes sense.

Since you asked, the drawback of just logging into an image, editing files, and installing software is that you can only reproduce that image by grabbing an entire file system.

When I use Docker, I create a git repository containing a 'Dockerfile', which is basically a series of shell commands to configure a machine. I also add copies of any configuration files I'll need, and use the Dockerfile to copy them onto the machine during setup.

This can be extremely fast in practice: Docker has a caching system which "runs" unchanged lines in Dockerfile by looking up a cached VM image, so I can often edit the Dockerfile and rebuild the image in a second or so.

This approach is really nice when I have to look at an image a year later and figure out how I created it, perhaps with the goal of upgrading to a new OS release or whatever. I just glance at the Dockerfile, change the base OS version, and re-run it.

I wonder if the UI could be improved. Like a "dockerfile creator" that lets you log in and make changes, then at the end lets you look at a list of changes or "history" entries and selectively pull them into a dockerfile.

http://blog.codiez.co.za/2013/09/setup-a-docker-container-wi...

sudo docker run -i -t ubuntu /bin/bash sudo docker commit [container id] [tag]

Is that what you are looking for?

docker commit produces an opaque container image that's not documented or repeatable.

"docker diff [container]" will show you what files have changed since the image was started, and you can then "docker cp" files out of the container to the host.

But for my part I tend to just layer changes piece by piece. So e.g. I have a docker image that's a "base development" image, and create separate docker containers to run each web app I'm experimenting with, with a shared volume with a separate docker container that I ssh into and run a screen session in with the code/git repositories. In effect that means that bringing up a new docker image for a new project is at most a couple of lines of change unless the project has particular/different dependencies.

I'm not sure there is truly such a thing as a "one-off" in the sense you're using it. What happens when your server dies (e.g. catastrophic disk failure, etc)—how do you get your service set up again? If your answer is basically "I re-make it from scratch", then Docker will save you a lot of time, no matter what scale you're operating at. Every Dockerfile is effectively a reproducible build; you'll never need to wonder what dark magicks brought the current state of the system about, because they're all explicitly documented right there, starting from a known-to-its-very-SHA base-image.

If you ever heard some of Heroku's propaganda surrounding "eternal applications", it also applies here: the stack you're deploying on will never update out from under you, because it's pinned by your Dockerfile. You can confidently move your app to any old server and it'll run exactly the same. One good analogy I've heard is that a Docker image is the reductio ad absurdum of a static binary: it doesn't depend on your system, just on itself.

Where do you put all of your data? Edit: specifically, ZNC keeps an archive of some channels for me. If I ran it in a Docker container, would the data be saved somewhere if, say, the power went out?

Just in case you missed it: you can "docker commit" the state of a container at any time (and "docker diff" to see what has changed on disk since creation). Would that solve your problem?

Docker is a superset of lxc.

But it behaves differently. For example, if you shut down a Docker container, changes are not saved.

In Docker 1.0 you can now use docker pause/unpause.

That's not true, and it has got nothing to do with the pause/unpause feature. You can see stopped containers with `docker ps -a`, and you can restart them.

Not anymore.

Although it still supports lxc, docker now defaults to libcontainer (https://github.com/dotcloud/docker/tree/master/pkg/libcontai...), it's own container implementation.

Yes. It is like configuration management [e.g. Chef, Salt]. It doesn't make sense to do for one off projects [unless you already have it as part of your workflow].

Like, I have a generic Flask container repository that I clone as a starting base...change the domain name(s) on the config file, app name, etc. Then just copy things into the src/ directory and spin it up.

For me, this is really easy to spin up a new flask project. I do the same with golang [sort of, it really just downloads a binary+config file into a container that has all of my logging, etc setup].

The problem with doing LXC is where you have inheritance chains like this:

[Base Container of Ubuntu + Logging + Monitoring] -> [Base Flask Container] -> [Flask Container for Website X]

[Base Container of Ubuntu + Logging + Monitoring] -> [Base Flask Container] -> [Flask Container for API for Website X]

[Base Container of Ubuntu + Logging + Monitoring] -> [Container for Mail Service]

[Base Container of Ubuntu + Logging + Monitoring] -> [Container for Kibana]

[Base Container of Ubuntu + Logging + Monitoring] -> [Container for Redis-Cache]

etc.

Tbh, I think that is what docker really fixes. The ability to easily inherit containers so you only have to make changes in one spot to change all of them.

My bet? Docker increasingly is something that will be used even if the user don't know it.

Builds, as an example. This was my first introduction to Docker - scaling Jenkins worker nodes was allocating yet another VM. As you scale to tens or hundreds of build slaves, you realize utilization across the cluster is down. Scaling builds with Docker was simple and efficient to implement and it allowed me to drive utilization up dramatically.

Now there's 15 different companies, integrations, and tooling surrounding this space to make that process repeatable to the masses. Amazing.

We're seeing the same thing in the orchestration, paas, sdn, iaas, YOUR ACRONYM OF CHOICE, and one common theme is that ecosystem is standardizing around Docker. That's pretty powerful.

I have few questions about docker

1. When I need to move application between different systems running Ubuntu, Debian etc, now I use Virtual box. Can I use Docker now on?

2. A quick reading about docker tells me that instead of running a guest OS as in Virtual box docker only holds application related things. Then how could it handle deployment between Debian Squeeze and Ubuntu 14.04. I mean old and new Linux version

3. Compared to virtual box how easy it is to install and use Docker

4. Can you please tell some places where you people use docker

5. How many of you have migrated to Docker from virtual box and related things?

Disclaimer: Noob detected :)

Does the complexity of using containers bother anybody? It just seems like this adds so many more attack points by removing the virtual machine which was a good way to organize services.

> Does the complexity of using containers bother anybody?

It doesn't seem all that complex; sure, its (in the typical cloud use case) another level of organization, but done right it should actually simplify organization and deployments.

> It just seems like this adds so many more attack points by removing the virtual machine which was a good way to organize services.

Containers are different than VMs, but using them doesn't mean "removing the virtual machine". Particularly in the use cases that Google is embracing (e.g., on a cloud platform where the containers are for use on VMs.) How, specifically, does it add "attack points"?

Increasing parallelization has been done in evolutionary steps for the last 10 years. This is a great step, but I'm sure your amazons and rackspaces of the world will soon be building docker++ (and docker really is just ec2/AMI++)

Rackspace and Amazon are heavily involved in the Docker ecosystem.

Reply on news.ycombinator.com