It doesn't really make sense to use Docker without some sort of distributed scheduling/orchestration system like Kubernetes, Docker Swarm or Mesos.
I think the first few years of Docker were just pointless but I genuinely feel that it is all starting to come together now.
Docker 1.12 looks promising on the orchestration front - Hopefully they will back all these new features with thorough documentation (which I found lacking in Swarm when compared to K8s).
It looks like Swarm is just following in K8s' footsteps and maybe even improving things along the way.
Based on what I just read, it looks like Docker Swarm might finally be a serious contender to K8s.
Yeah I see where you are coming from. First time I heard about Docker, I thought it did what NixOS does, but it doesn't really.
With that it is really weired to read all complains and issues about volumes management. It seems Docker just picked up wrong model for their storage. Then they try forever to fix impedance mismatch between that model and how people want to use Docker.
I think the biggest issue is around the `-v` syntax (and the corrosponding API). In the UX we call things a volume which are actually a mount (which happen to use a volume). I like the way we have implemented this in the new services API.... `docker run --mount type=volume,source=important_data,volume-driver=foo,volume-options=...`
For example, it is very useful to have a shell view of all volumes for the application to poke inside or debug issues. Surely I can do docker run ubuntu and mount all relevant volumes under /mnt there, but that is not user friendly.
Or consider that during development or debugging I want to transfer the state to another machine. Why it is not possible to tar all application volumes with single command and restore them in a new place again with single command? I.e, something similar for docker save/load, but for volumes?
All those issues are trivially solvable if I use some directory on a file system for the whole application state and use subdirectories there for individual volumes for the containers and then pass those with -v to relevant containers.
check out weave for managing networking, IMO it's much easier and I'm not even talking about the docker plugin. just use weave without the plugin and it works amazingly well.
for the configuration aspect, just use dockerfiles and a system like mesos and marathon for deploying production.
you can then write simple restful API commands to launch different environments with marathon.
build all your docker containers to act as single server instances, i.e., spark on one container, SQL on another.
tie in a default install of cadvisor on all mesos clients and you can have easy monitoring as well.
Back in my previous project using Cordova, I made the Docker workflow work adequately by offloading builds to web-based build services like PhoneGap Build and Ionic Package (building only a thin live-reloading wrapper app that points to a configurable IP:port for development). But my new project uses React Native, and I haven't found any similar build services for React Native apps (or even just regular native apps) yet. Anyone have any suggestions?
On the other side devs were asking for jenkins for that process. We run also Jenkins on Docker but it's cumbersome as it doesn't allow declarative configuration. There are dozen of plugins out there but it doesn't change the fact that the outcome is ressource hungry and slow. For now we are actually optimizing using Job-DSL plugin. We reached our target, but it was extremly time consuming and pain. For a new project I'm in, maybe we go to replace jenkins with concourseci or drone.
Imho a PaaS doesn't solve processual versioning and testing requirements. For that you still need further tools - with or without docker.
I love Concourse. Look me up if you have questions.
That's bundling an CP application into seemingly AP infrastructure (from what I gleaned from their docs). Completely the wrong tool for the job. If the application was designed for that it would have recovered. In your defense, they have nothing to warn about that in their documentation.
Not saying that's a good idea, but it is getting closer.
You could, for example, have a node that's only for DBs that has volumes on it. You could then use DRBD on the host to clone that data to a secondary node. then in the event that node 1 dies swarm would bring the DB up on node2.
With the mesh network stuff they've added the endpoint would remain the same, so all your apps would need to do would be re-connect.
No, that isn't getting closer it's getting farther away. The whole point of containers is that they make the host machine completely fungible. If I can only schedule my DB containers on a specific machine then I might as well just run my DB on that machine and be done with it.
I'm saying that, if you want, you CAN make it guarantee that a container will be rescheduled to the same host. (or a controlled level of hosts).
I don't think it's docker's responsibility to solve database clustering.
And, as I said, I'm not advocating running a production DB in docker. But I can see a way that you may be able to.
> I don't think it's docker's responsibility to solve database clustering.
I don't know if I'd call it a responsibility. But Docker is obviously trying to expand their platform into more and more aspects of containerization. If they figured out persistence that would really set them apart, something that this product doesn't really do. It mostly just keeps them even with Kubernetes at best. And an imitation of it at worst.
100% agree with you there.
> It mostly just keeps them even with Kubernetes at best. And an imitation of it at worst.
Personally, when I looked at kubernetes, mesos, etc and saw the xml hell that i'd be living in and said "no way".
If i have to run something as complex as those to have docker then it's not worth it to me.
When I look at the new docker swarm stuff. For the first time I think that docker is a viable thing (assuming it all works). because I'm not adding additional complexity to get my docker simplicity.
Volumes are exactly what you should be using to have data that lives beyond the life of the container. Essentially, if you are writing to disk in your container, you should probably be writing it to a volume.
Volumes can be backed by any number of storage solutions, including from every cloud provider, various distributed block, gluster, ceph, etc...
Docker should _not_ be changing the way you run a database. If you need a highly available database, you don't use 1 database process and hope for the best, you use the built-in replication features available in just about every database product in existence today... database goes down, oh well you have a hot-standye ready.
That is, Docker is just a packaging system + a tool to spawn the deployed stuff in an isolated environment. Now, 3 years after, they've improved dependencies (previously done by docker-compose + hacks for using it in production).
That is, I use Docker only because its images are somewhat easier to build and deploy, compared to .deb packages or plain tarballs (and it's more OS-agnostic, since Docker Machine's available on Windows and OS X, so I can just build an image with dev. environment and don't care what OS is used). Doubt it's something more than this.
There is a technology for that already: it's called OS packaging. Docker does not maintain enough metadata to allow for in-place upgrades of individual components in an image (software and configuration lifecycle management). The best you can do with Docker is install lumps, you can forget upgrades. Docker is not a replacement for configuration management, and specifically, Docker is not a replacement for the operating system's software management subsystem.
Your missing the point of a container, you don't upgrade them in place
On SmartOS, when you provision a zone, you get a fully virtualized UNIX server, and you can apply a container to it by defining resource constraints, but that is both pointless and unnecessary there. Once you have a fully virtualized server provisioned from an image with the imgadm(1M) and vmadm(1M), it is only logical that you will want to service individual components via pkg_rm and pkg_add, rather than baking an entire image all over again, and redeploying it, all over again. It's the rule of modularity: "write simple parts connected by clean interfaces" [Kernighan-Plaguer], and it applies particularly well to lightweight virtualization.
Docker is just a quick-and-dirty way to have a full system image. It's somewhat simpler than building a proper .deb (.rpm, .tar.xz, whatever one distro eats) package, especially when there's a ton of dependencies from other packaging systems (PyPI, Ruby gems, npm packages, etc.)
Oh, and unlike with many popular OS packaging systems, with Docker can actually have multiple versions of the same "package" "installed" at the same time (that's my biggest issue with dpkg), but IIRC there are no built-in provision for the migration scripts (prerm, postinst - this sort of stuff).
Docker has support for a wide-array of storage solutions, everything from the various providers solutions (ebs, s3, gce storage, azure storage), to vsphere, netapp, scaleio, various distributed block solutions, nfs etc...
You should _not_ be changing the way you handle storage just because you are in a container. Use the tools you already have. If you need HA on a database, use the built-in replication service available on just about every database product in existence. If you really want distributed/shared storage, see the list above.
Containers are not black boxes.
There are often valid reasons to not do this. For example, MySQL does not guarantee consistency between a source and its destination. mysqlrplsync can do this as a bandage, but it's something extra you need to set up and configure.
Replication != backup
You mean exactly like zones handle it inside of SmartOS (;-))
The ZFS storage pool is abstracted away from the zone the virtual server runs on, and since it is persistent, it's completely out of the way, JustWorks(SM).