Docker 0.7 runs on all Linux distributions
blog.docker.io
blog.docker.io
To remove the hard dependency on the AUFS patches, we moved it to an optional storage driver, and shipped a second driver which uses thin LVM snapshots (via libdevmapper) for copy-on-write. The big advantage of devicemapper/lvm, of course, is that it's part of the mainline kernel.
If your system supports AUFS, Docker will continue to use the AUFS driver. Otherwise it will pick lvm. Either way, the image format is preserved and all images on the docker index (http://index.docker.io) or any instance of the open-source registry will continue to work on all drivers.
It's pretty easy to develop new drivers, and there is a btrfs one on the way: https://github.com/shykes/docker/pull/65
If you want to hack your own driver, there are basically 4 methods you need to implement: Create, Get, Remove and Cleanup. Take a look at the graphdriver/ package: https://github.com/dotcloud/docker/tree/master/graphdriver
As usual don't hesitate to come ask questions on IRC! #docker/freenode for users, #docker-dev/freenode for aspiring contributors.
In the future the drivers should support using actual lvm-enabled devices if you have them.
The boundary between "kernel" and "libraries like libc" is very stable and doesn't change often. That means that often, the kernel distributed by Arch can work reasonably well in an Ubuntu system, and vice versa.
With that in mind: The "ubuntu" image ships the "ubuntu-glibc" and "ubuntu-bash" and "ubuntu-coreutils" and so on, but they continue to work on your Arch host because the system calls don't ever change.
You can't link (say) ubuntu-glibc into arch-bash though, which is why containers are built off of a "base ubuntu image" in the first place.
Containers come with their libraries though; you don't have to "add" anything. You'd just apt-get it within the container and it would pull down its dependencies.
What happens if you put a 64bit executable in a container they try to run the container on a 32bit machine? Or a Rasberry/Arm device?
My goal is having a team of developers use Docker to have their local development environments match the production environment. The production environment should use the same Docker magic to define its environment.
Is the idea that developers define their Docker environment in the Dockerfile, and then on app deployment, the production environment builds its world from the same Dockerfile? How does docker push/pull of images factor into that, if at all?
Or is the idea that developers push a container, which contains the app code, up to production?
What happens when a developer makes changes to his/her environment from the shell rather than scripted in the Dockerfile?
What about dealing with differences in configuration between production and dev? (Eg. developers need a PostgreSQL server to develop, but on production, the Postgres host is separate from the app server - ideally running PG in a Docker container, but the point being multiple apps share a PG server rather than each running their own individual PG instance). Is the idea that in local dev, the app server and PG are in two separate Docker containers, and then in deployment, that separation allows for the segmentation of app server and PG instance?
I see the puzzle pieces but I am not quite fitting them together into a cohesive understanding. Or possibly I am misunderstanding entirely.
Docker lets you build an environment (read: put together a bunch of files) for you to run an app in. It also has other features, like reducing space if lots of your apps [on the same host] use the same docker images, and networking stuff.
> My goal is having a team of developers use Docker to have their local development environments match the production environment
You run a container. You use images to distribute files. A Dockerfile is a loose set of instructions to build the images.
The basic idea is that, any single program that you want to be able to run anywhere, you make into a container. You can run it here, you can run it there, you can run it anywhere. The whole "running it anywhere" concept comes from the idea that all of your containers are created based on the same images, so no matter what kind of crazy mix of machines you have, your containers will just work - because you're literally shipping them a micro linux distribution in which to run your application. And since all the applications are running in isolated little identical containers, you can run as many of them as you want, independent of each other, in whatever configuration you want.
You'll have a PSQL container and an App container, and you'll manage them separately, even if they share images - the changes they make get saved off to a temp folder so they don't impact each other. Your environment stays the same only as long as the containers and images you're using are the same.
There will always be differences between development and production. You have to focus on managing those differences so you have confidence that what you're shipping to production actually works. The only thing Docker really does there is make sure the files are basically the same.
A developer will use a Dockerfile to build the main dependencies that you need, and then save an image. Then to deploy code (or other frequently changing data), you pull that base image, add or update whatever else is needed, and save it again. (You can add layers to an image as often as desired.)
Then this saved image is pushed into a production environment. It seems like configuration data is often saved in images, so they require zero configuration. However, you can pass as many command line parameters as desired to run the image in production, so you can keep your configuration separated (like http://12factor.net/config) if you want.
It's only a matter of time before linux namespaces get more scrutiny and people grow more comfortable with them as a first-class security mechanism. It took a while for OpenVZ to get there, but not there are hosting providers using it for VPS offerings.
On top of namespacing, docker manages other aspects of the isolation including firewalling between containers and the outside world (currently optional but soon to be mandatory), whitelisting of traffic between certain containers, etc.
See my other comment in this thread for a more accurate answer.
That's the idea. Docker is based on LXC which provides configurable isolation between process groups. So, you could run your http in one container, your database in another, and your application server in a 3rd and have reasonable confidence that they can only talk to each other the way you want them to.
The other opinion (given that you don't enable -privileged mode) is that lxc namespaces _do_ protect your host and other containers from being exploited due to compromise of a contained process, and cgroups _do_ protect your precious i/o and cpu cycles from being fully consumed and deadlocking your whole system when a rogue container decides to start infinite looping.
The competent answer of course is "try it", you should know that there are ways your processes can be compromised, and you should check if the exploits you know about result in the same or different privilege escalations when you apply them to a service hosted within a docker instance. That being said, you can't really prove a negative...
Can this reduce the time it takes me to put up and Ubuntu installation on Digital Ocean?
Is this more for larger companies ?
Docker can help you streamline your development and deployment process across all the machines, from the development laptop to the production server. The result should be that your development environment is easier to setup and replicate, more similar to production, and easier to customize as you evaluate new tools, hosting providers and software components.
I recommend checking out the "Getting started" page, it has a great online tutorial: http://www.docker.io/gettingstarted/
You should also browse through this list of real-world use cases. http://www.docker.io/community/#What-people-have-already-bui...
I hope this helps!
Edit: The website/tutorial is open source too. https://github.com/dotcloud/www.docker.io
yum --enablerepo=epel-testing install docker-io
PS: make sure you have "cgconfig" service running
Any info on how it sets up networking? I see -b none in the startup... Is there some other tool that does it on EL6?
Do you know where devel talks for this rpm are happening? I did not see any mention of it in the EPEL list archives.
Could you post a detailed instruction?
Thanks a lot!
For disk IO, as long as you use data volumes [1] for performance-critical directories (typically the database files), then overhead is also zero. For other directories, you incur the overhead of the underlying copy-on-write driver. That overhead varies from negligible (aufs) to very small (devicemapper). Either way that overhead ends up not mattering in production because those files are typically read/written very unfrequently (otherwise they would be in volumes).
[1] http://docs.docker.io/en/master/use/working_with_volumes/
You might be able to run docker on Android though, but you may need to compile a kernel with containers (namespace) support as I don't know that Android ships with them (I forget).
docker newb here. Can I easily put my own software in it? I've got this c++ program that has a few dependencies in ubuntu.
Documentation link: http://docs.docker.io/en/master/use/builder/
Can I just hand it a .tar.gz and say, put it in /usr/asdf and let it run? What about python scripts? Maybe I just give it a location to an RPM? like in this document?
Here are 2 examples:
https://github.com/steeve/docker-opencv https://github.com/shykes/docker-znc
I wonder if this would be good for (large) embedded systems. Thanks for the tip.
I mean, it'd be neat to be able to do a "pull" of diffs from one image into another related image. Merge branches and so on. I don't know, possibly this would be just too unreliable, but I would have previously thought that what docker is doing right now would be too unreliable for production use, and lo and behold we have it and it's awesome.
http://blog.bittorrent.com/2013/10/22/sync-hacks-deploy-bitt...
http://blog.docker.io/2013/07/docker-desktop-your-desktop-ov...
There has been discussion of taking the git analogy further. We actually experimented with a lot of that early on (https://github.com/dotcloud/cloudlets) and I can tell you it's definitely possible to take the scm analogy too far :)
I do think we can still borrow a few interesting things from git. Including, potentially, their packfile format and cryptographic signatures of each diff. We'll see!
Here's a relevant discussion thread: https://groups.google.com/forum/#!msg/docker-user/CWc5HB6kAN...
So much for feature #7. Documentation should be part of the development/release process
Sorry about that. One more lesson learned on our quest to ultimate quality!
Is the warning "This is a community contributed installation path. The only ‘official’ installation is using the Ubuntu installation path. This version may be out of date because it depends on some binaries to be updated and published." still true? Fedora also has this warning (and no instructions). The "Look for your favorite distro in our installation docs!" link does not give me up-to-date instructions for any of my favorite Linux distros. I can't even see where in that installation documentation it says how to install from source code on generic Linux. What am I missing? (Of course I can get the source code and build it, but I want the documentation to be great :-D)
I guess we could reference that it the install docs.
I see lots of people are getting some generic Docker questions answered in here, and want to ask one I have been wondering about.
What is the easiest way to use dockers like I would virtual machines? I want to boot an instance, make some changes e.g. apt-get install or edit config files, shutdown the instance, and have the changes available next time I boot that instance. Unless I misunderstand something, Docker requires me to take snapshots of the running instance before I shut it down, which takes an additional terminal window if I started into the instance with something like docker run -i -t ubuntu /bin/bash. I know there are volumes that I can attach/detach to instances, but this doesn't help for editing something like /etc/ssh/sshd_config.
This is great for development or playing around with something new, but the best practice for creating a reusable image with your custom changes would be to write a Dockerfile which describes the steps necessary to build the image: http://docs.docker.io/en/latest/use/builder/
You're right, I misunderstood what docker was doing when shutting down the container. Seems like I can start and reattach just fine. Here is an example workflow for anyone curious:
root@chris-VM:~# docker run -i -t ubuntu /bin/bash
root@0a8f96822140:/# cd /root
root@0a8f96822140:/root# ls
root@0a8f96822140:/root# vim shouldStayHere
bash: vim: command not found
root@0a8f96822140:/root# apt-get install -qq vim
...<snipped>...
Setting up vim (2:7.3.429-2ubuntu2) ...
root@0a8f96822140:/root# vim shouldStayHere
...Not exactly necessary, but I added a line to the file so I could identify it...
root@0a8f96822140:/root# exit
root@chris-VM:~# docker ps
ID IMAGE COMMAND CREATED STATUS PORTS
root@chris-VM:~# docker ps -a
ID IMAGE COMMAND CREATED STATUS PORTS
0a8f96822140 ubuntu:12.04 /bin/bash About a minute ago Exit 0
root@chris-VM:~# docker attach 0a8f96822140
2013/11/26 10:29:41 Impossible to attach to a stopped container, start it first
root@chris-VM:~# docker start 0a8f96822140
0a8f96822140
root@chris-VM:~# docker attach 0a8f96822140
ls
bin boot dev etc home lib lib64 media mnt opt proc root run sbin selinux srv sys tmp usr var
root@0a8f96822140:/# cd /root
root@0a8f96822140:/root# ls
shouldStayHere
root@0a8f96822140:/root# cat shouldStayHere
Hello World!
root@0a8f96822140:/root#
So, if I did some heavylifting to set something up and wanted to keep this as a base for later work, now I would do e.g.
docker commit a8f96822140 <some identifier>I prefer to never keep anything important in a stopped container (for very long) without committing it back to an image, and I don't like dealing with numeric ids.
Recently (it looks like you don't have this change yet) docker added the automatic naming scheme giving every container a random name of some "color_animal" pair which I think reinforces the point, stopped containers are not a place to store meaningful/persistent state information for very long.
This mishmash gets run almost every day on my docker hosts to clean up after terminated experiments:
docker ps -a|egrep -v 'ID|Up'|awk '{print $1}'|xargs docker rm
Beware, it will delete all of the stray containers you've ever created before that are now stopped!
For example a common pattern is to create a placeholder database container with reference dara as a volume, but no actual process running and no network ports. Then successive versions of the database container are started on the side, with shared access to the placeholder's volume. Later you might run a backup container on the same volume. In other words you can point to a particular dataset as a container of its own, separate of the various applications which might access it. All of these interactions are visible to docker so it can authenticate, restrict, log, or hook them in all the standard ways.
In fact internally images and containers are stored side-by-side. In future versions we are going to accentuate that similarity.
The changes you mention sound nice. It's no surprise I don't have them:
root@chris-VM:~# docker version
Client version: 0.5.3
Server version: 0.5.3
Go version: go1.1
It was the easiest VM I had access to at the moment of posting. I should update the docker in there.I have used docker ps -a | awk '{print $1}' | xargs docker rm a couple times to clean up after playing around. I was slightly annoyed that it tried to docker rm a (nonexistent) container with the id ID. Thanks for reminding me to throw a egrep -v 'ID|Up' in front of awk.
https://github.com/dotcloud/docker/pull/2837
tl;dr They are going to pick a new pair of things for every major release of Docker. This is meant to let you keep track of more containers over a long time. Apparently you are in fact meant to keep them around if they're still in working order, and remember them "by ID" or by name.
Your question is very near to me, I published a teeny-tiny script called 'urbinit'[1] for doing exactly what you are asking. It's meant to use with urbit, where you start by creating a 'pier' and use it to begin 'ships'. The ships must not be discarded (unless you don't mind having no real identity, and many parts of urbit require you to maintain a persistent identity). If you re-do the key exchange dance again from a fresh pier to identify yourself as the same 'destroyer' (having discarded the previous incarnation), you wind up as a perfectly viable ship with packet sequence numbers that are out of order and can't really receive any messages or acknowledge any packets that are waiting to be delivered in the correct order from your neighbor ships.
Anyway, urbinit is a manifestation of the fact that Docker doesn't really have anything to help you deal with this (very common and not at all unique to Urbit) problem. Docker offers that you can use a volume... make the state you need to be persistent part of a volume. Unfortunately this means it won't be present in your images; or you can do roughly what urbinit does, which is a very simple and classic problem all Computer Scientists past year 1 should already know.
Launch the container in detached mode (-d) and save the ID. Create a lock file. Urbinit won't do step 1 if there is already a lock file created that was never cleared. This is to prevent you from launching the same ship twice at one time with two addresses, which is very confusing to the network.
Attach if you need to interact through terminal, or however you want, affect the container by sending packets to it (eg. login via ssh and do stuff). Terminate your process or let it end when it's ready to end naturally, or the power goes out. If you're using urbinit (and the power did not go out), this is the point where the commit process comes along and commits your container ID back to the named image that it was created from, simultaneously clearing the lock created in step 2.
If you're doing this with several different images on one host that are meant to run simultaneously, a nice trick is to name your lock file after the actual docker image that's used in the commit step.
Instead of lifting the limit for some drivers first (which would mean some images on the index could only be used by certain drivers - something we really want to avoid), we're artificially enforcing the limit on all drivers until it can be lifted altogether.
If you want to follow the progress of the fix: https://github.com/shykes/docker/pull/66
(This pull request is on my personal fork because that's where the storage driver feature branch lived until it was merged. Most of the action is usually on the main repo).
For aufs, I have no real data, but I assume that a 42 layer image is somewhat slower than a 1 layer image.
The fix for going to > 42 layers is to recreate a full image snapshot (using hard links to share file data) every N layers, so the performance would be somewhere between the performance between 1 and 42 layers depending on how many AUFS layers you got.
Some background: Using 0.65 it took me several days to develop a Docker image with manually compiled versions of V8, PyV8, CouchDB, Flask, Bootstrap and some JS Libraries[1]. With no 42 limit it wouldn't have taken so long since I would have had way more caching points - however I'd also be afraid about performance.
[1] the image is available on the repository as ategra/xforge-dependencies. I can upload the Dockerfile to github if anyone is interested.
I don't think there will be any performance problems with deep layering of images, either on dm or ads.
Of course if you had infinite memory and those kernel data structures were designed to never overflow, then you would have a performance problem on your hands instead of a kernel panic :)
I was pretty sure that the requirement for AUFS would stick for a long time -- I was resigned to use a special kernel. But again, you folks surprise me!
You guys just rock!
I can't find the info in the docs.
It's not clear to me how I can benefit from Docker given my setup above. Any comments?
Chef is based on re-running the same commands with various different options depending on the environment, and even without any thing in the cookbooks/attributes/environments changing, Chef still cannot guarantee that this run will produce the same results as a run that happened yesterday, simply because it isn't like an image.
Imagine spinning up your db instance vm, your web tier vm, your load balancer vm... etc.
Unless you have a ton of ram it isn't going to happen. With docker you can run containers that mimick a very very large infrastructure on your laptop.
Let me ask a direct need I have, would docker allow me to use newer c++ compilers on redhat so I can code in c++11?
It's just s/distrubtions/distributions/, obviously.
Is there any alternative for separating apps on a single VPS?