Docker in Production: A retort | Hacker News Reader

Docker in Production: A retort

patrobinson.github.io

133 pointscrymer11121 comments

I've never used Docker, or containers, but I read about things like "Breaking changes and regressions ... a well documented problem with Docker" and "Can’t clean old images ... a well known issue" and it just seems to me like a crazy thing to try to use and depend on this thing/company. Bluntly put they seem like children.

So nevermind a retort, what I would like to see is a sane, sensible "business value" cost/benefit, pros v. cons breakdown of just what the heck you're actually gaining (and losing) using Docker vs. some other architecture/methodology. Because absent that it's all just hype and kool-aid drinking in my opinion.

What would help with the above is if people would document what they are doing with Docker that works, because either they are hurting but not realizing it, or the author of the article is just "doing it worng" and whining about it in public. What is really going on with Docker, et. al.!?

I can tell you one place I've found them very valuable: development systems. The longer I've been in the field, the more I've learned that polluting my dev box with globally installed tools makes life painful down the road. Docker provides a nice way (especially now that they have docker exec) to spin up environments, or set's of environments with tools/stacks and not touch the host. It's lighter than a bunch of vm instances and easier to orchestrate on a small scale than most tools I've found. Docker for Mac makes that even nicer!

Completely agreed. My workplace adopted Docker for local development first, before using it anywhere else. Aside from the "volume mount wiping the `node_modules` folder that was installed in the image but doesn't exist in the host on first boot" problem, it's been an absolute dream.

We had some issues with docker-compose for our use-cases, such as persistent database even when recreating every container from scratch, so I hacked on some stuff that become "dup" [0] (nee. docker-up)

Ps. it's written in Nim, which was a lot of fun!

[0] https://github.com/girvo/dup

Assuming you're also running your db in dockrr, why didn't attaching docker volumes work for you for database persistence ?

Because docker-compose had a very bad habit of destroying and rebuilding the volume-only container, unfortunately! This was of course back when running Docker on macOS was somewhat painful, so mounting the DB data directory directly as a volume into your host would nearly always fail to work successfully due to screwy permissions errors (it's quite a bit better now with Docker for Mac)

So, some dodgy code and a JSON file later, and our team uses the `dup` CLI to run everything with exactly zero hitches thus-far, and we've been using (across it's two incarnations) for... 18 months now?

I find using nix as a package manager to be a better solution. Write a nixfile that have all the tools you need for a script, and you're fixed.

Docker is a slow, blunt tool for just containerizing dependencies IMO.

A slow, blunt, cross-platform tool. So while I agree, I don't think nix makes a good replacement for docker.

Cross-distribution, to be precise. You can't run amd64 image on RPi for instance.

Cross platform. Runs on Linux, Windows, and the Solaris port is being merged. There was a FreeBSD port but it was alas not upstreamed and needs some work.

No it does not change processor architectures for you.

It's great for development if it wasn't 4 to 5 times slower on Mac: https://www.reddit.com/r/docker/comments/59u1b8/why_is_docke...

Still feels several times faster than using an option like Vagrant. (haven't found a container that didn't cold start in less than 5 seconds; 3+ minutes for common Vagrant setup is pretty common)

(Having read the Reddit thread for a while)

Macs don't appear to have support for containers which is what Docker is. Well that is probably bollocks because they are *BSD based and I know that FreeBSD at least has a form of containerization and I seem to recall that the whole container thing was invented on a BSD - "jails".

So Docker can't run native on iStuff. You have to run it within a Linux VM. I gave up on Googling for "apple mac jail" 8)

The reason why Docker runs slowly on Macs is because it is running under emulation within a VM.

Unfortunately, Macs are not cool enough to run Docker, so there 8)

Cheers Jon

XNU (the macOS kernel) may have BSD origins, but it doesn't have features like cgroups and process namespacing that the Linux kernel provides and the Docker runtime relies on.

Nor does it have FreeBSD jails.

Agreed! So far I've used Docker only on Dev systems. I'm still waiting to be convinced that there's sufficiently higher productivity and value to be had by introducing it to Prod.

Also for CI in the same vein.

> seems to me like a crazy thing to try to use and depend on this thing/company

Always beware of perception bias or whatever the right term is for this. When docker was new it got tons of hype and people were talking about how great it was. Now it's no longer the hot new thing so it's just another tool many people are using quietly and successfully. It no longer warrants much positive discussion. The people using docker in production successfully don't feel compelled to write articles about it working for them.

I'd say the pros/cons of docker (or containers in general) are similar to the pros/cons of buying hand made vs. mass produced goods. The hand made goods will probably be higher quality but they take more time and skill to produce so they're more expensive. The mass produced goods will probably be lower quality but will work fine and at least offer consistent/reliable level of quality. They take less time and skill to produce so they'll also be less expensive. In the context of using Docker in production this translates to it being very good for solving 'off the shelf' types of problems quickly and reliably. It's essentially the software/systems version of an assembly line. If you don't need an assembly line then docker may be worthless or even a total waste of time. If you do need an assembly line it's the best option currently.

You're taking the wrong assumption here.

There aren't articles about docker in production because barely anyone adopted it for major production environments [yet]. And for the few who are early adopters, there wasn't enough time to write articles.

As someone considering using Docker for production infrastructure I'd love to hear from a docker expert what their thoughts are on the following observations:

* Docker encourages fully disposable infrastructure

* Docker containers can be more secure then traditional environments

* Docker provides for indempotent environments agnostic of hardware concerns etc

* Container management software (Kubernetes, etc.) makes Docker much more powerful/useful and is only going to get better

Specifically, I'm curious if my observations are correct, and if they are why they are correct and why are they SO much better (vs current docker alternatives including non-container based approaches).

Author from the original article here.

>>> * Docker encourages fully disposable infrastructure => I'd consider that incorrect and/or misleading

Docker is meant to manage disposable (i.e. stateless) applications. They are expected to be killed/stopped/restarted at any time, no persistent state, no data.

An application is either disposable OR not disposable by its design, docker doesn't change that. (For instance, don't try to dockerize databases).

Docker gives you a quick, nice and easy way to package your applications with dependencies (as Docker image) AND deploy them. [This is valuable]

It's especially great for development and test environments because you can run many containers (i.e. applications) on a dev/test machine, without dependency/deployment hell. [This is very valuable]

>>> * Docker containers can be more secure then traditional environments => No. Just No.

Docker has nothing to do with security.

>>> * Docker provides for idempotent environments agnostic of hardware concerns etc

Docker gets to ignore the hardware. You do not.

You get to buy, setup and manage servers. Maintain a full environment that can run docker.

It can be an undertaking full of dangers, as told by this article and mine :D

>>> * Container management software (Kubernetes, etc.) makes Docker much more powerful/useful and is only going to get better

Docker allows to package and deploy an application. It doesn't do anything about infrastructure at all (i.e. machines, load balancers, DNS, discovery), which you still have to manage and it's a HARD problem.

The orchestration systems is supposed to help you manage all of that. As a result, it will be able to automatically kill/restart/move/add containers across a cluster and make the necessary [physical or virtual] infrastructure changes.

Container + Orchestration will be a [r]evolution to manage entire giant fleets of hosts and applications. [This will be infinitely valuable (allowing me to automate the work of 10 people, all freeing all my time to enjoy unlimited free drinks and foods)]

IMO: In the current state of affairs. Having docker without orchestration is like having cars without roads.

> For instance, don't try to dockerize databases

Host volume mappings. Problem solved.

I've been using them for over a year, with all kinds of different databases, postgres, mysql, mongodb, in dev, ci, and production, never had a problem. It gives you a nice separation of what is and what isn't part of the application state, and makes backups a breeze.

Right. This "don't Dockerize databases" thing is truly just bizarre. It almost in a way seems intentionally antagonistic to make Docker seem less capable than it is. After all, who wants to use something that only works for some applications? And the fact of the matter is that Docker is just a fancy way to run a process. Bind-mounted host directory volumes are one of the most fundamental features available for Docker. The truth of the matter is that Docker can be a complex beast. And what ends up happening is that often people only understand a small part of Docker features they're using. And then they complain about problems with Docker when what they really need to do is go read the documentation. Not all problems with Docker are like this, of course, but the vast majority of complaints that I see (and I see a lot of them from being part of the open source community for over three years now) come from people that don't fully understand the features of Docker they're trying to use. Getting Docker going is dead simple, and people fall into the trap of thinking that's all there is, when Docker is a damn deep technical subject.

Is this what you're referring to?

https://docs.docker.com/engine/tutorials/dockervolumes/

>Having docker without orchestration is like having cars without roads.

Docker's stated purpose is to be the "shipping container" abstraction for server-side code.

Shipping containers are not much more exciting than any other kind of box (VM, jail, whatever) in isolation. Things get more interesting when every port has container-optimized cranes and there are thousands of container ships competing for your business.

Thinking about it i seem to recall that for a while all the hoopla was about VMs that could migrate between hardware to both better utilize existing hardware resources, and allow for more uptime.

But then a while later all the buzz was containerization, with Docker as the poster boy. This, i suspect, in that rather than have the overhead of a VM and such, you could spin up a container instance over there, take it down over here, and achieve much the same result. This in particular if the content of your VM or container was talking to a storage backend anyways.

In my context it's because big data became a thing. Nobody wants to pay the virtualization IO overheads anymore.

That said, I've found myself swinging back towards preferring VMs - with docker-like infrastructure made available. The runv project (https://github.com/hyperhq/runv/) is an interesting and functional take on Docker-with-VMs since I spend way too much time wanting to do privileged things in containers.

> don't try to dockerize databases

Why ever not? There no less reason to do so in a container - and all the usual benefits still apply.

>>> * Docker encourages fully disposable infrastructure

I just want to tack on the upgrade rollback situation. A container looks a lot like a process from the host os point of view. You can bring up the new release and leave the old release running on the same machine without much difference in load. The load just shifts to the new container. With a VM approach, you kind of need both vm's for the whole upgrade window.

Depending on your orchestration, blue/green deployment is pretty straightforward.

There can't be two containers running that listen on the same port. The rolling upgrade will need to kill the first container to start the new one.

I may suppose that you talk about [CPU] load, because you are running computations/processing applications and not webservices.

Well, this is a problem that's been solved [0]. I've been using nginx-proxy for a couple of years now, in production, without any issues.

[0] https://github.com/jwilder/nginx-proxy

Don't use static ports. Pick a random one and update the load balancer (Haproxy, Traefik, Nginx) to point to the new one. If the rollout fails, you can always roll back by pointing back the old one.

FWIW, Kubernetes solves this out of the box.

> No. Just No. > Docker has nothing to do with security.

Docker can totally reduce the risk of in-container services being exploited. It has sane defaults, and makes it easy to distribute other restrictions with the application (seccomp profile, apparmor settings, etc).

I'm not sure if I qualify as a docker expert, but I feel like I need to comment on these...

Firstly, I think it is correct that docker encourages disposable infrastructure, because your docker setup basically must be able to recover from the loss of a container. I think it would not be sane to expect your containers to "just keep working".

Regarding security, I don't think Docker provides much other than the illusion of better security at the moment. At least on Linux containers don't add security that you can't get with traditional services.

I suppose if you use authenticated images you can at least be sure that you're running the code you think you are, but then again package managers have had signing for ages.

You're always going to have to test your software in the environment you will be deploying it in, because driver bugs and weird interactions still exist. Docker doesn't really let you ignore the hardware

I have no opinion of kubernetes and such at the moment.

What docker does grant is an easy method of "packaging" software... Pretty much anyone can just throw everything together in a big bundle and distribute that. For certain use-cases, this can give a development team much more speed, since setting up a workable environment is much faster. There is also a nontrivial amount of software that simply does not bother with OS packaging where containers can help you deploy it in a more controlled fashion.

'I think it would not be sane to expect your containers to "just keep working".'

Why?

As SysAdmin, I'm going slowly but surely to Docker. You have to follow good practice such as using devicemapper which is truly not by defaults even it CoreOS.

For sure for developer it's a big thing, for SysAdmin and people thinking about the datas it's more a pain in the ass.

Anyway, no way to go back. This is the future Jack.

On the last DockerConf you could ear ADP talking about their success with Docker, but obviously their there to promote Docker. ;)

* Docker encourages fully disposable infrastructure

Absolutely. You will get the most out of it like this. You can run pets, and treat containers VMs but I don't think you will get all the advantages.

* Docker containers can be more secure then traditional environments

Out of the box you get isolation from namespaces (not being able to see the host filesystem or network), resource limits via cgroups, a default seccomp profile that restricts some system calls that could be used in exploits, and more. You can set all these defaults yourself but the majorty of people do not. So there are definit benefits.

* Docker provides for indempotent environments agnostic of hardware concerns etc

Most applications do not talk directly to hardware, the default container does not directly present hardware interfaces, yes.

* Container management software (Kubernetes, etc.) makes Docker much more powerful/useful and is only going to get better

It does help to have software to manage programs across multiple computers, for scaling and redundancy, yes.

I think old school sysadmins care way more about system stability and new school devops guys are focused on cost and the speed of the development cycle. It really doesn't matter if you architect your systems properly that docker is unstable or your services crash as long as you have effective monitoring and self healing systems.

I'm both or rather I do things like this:

* Use modern tools to spin up a dev env * Determine how to monitor and maintain dev and deploy * Test dev env * Determine, based on dev and a deep risk assessment how prod will look * Create prod env * Determine how to monitor and maintain prod env and deploy (based on your test) * Test new prod env * Open up prod env to end users

It's not bloody rocket science - it's Engineering and the steps above hide loads of steps and feedback loops such as a safe upgrade method eg snapshot/clone VM, test, release.

If your product is important to you then you take it seriously and treat it right!

Cheers Jon

PS Did I mention documentation?

Docker founder here.

I keep reading articles stating that "the Docker API changes with every release", but the assertion is never backed by any specific examples. Has anyone here encountered an actual breaking change? If so, I would appreciate you sharing the specifics so we can fix it.

Docker is by no means perfect:

- I remember that in 1.10 the switch to content-addressed registries meant that older clients could not pull by digest (but all other commands, and even non-pinned pull, still worked). This was not an accidental breaking change: it was the result of a difficult tradeoff. In the end we decided that the benefits of the new content-addressed model outweighed the inconvenience. To guide our decision we used data from Docker Hub to assess how many clients would be affected. I forget the exact number but it was a very small minority.

- And in 1.12 we got bitten by a change in how Go 1.6 processes HTTP headers (it became more strict and thus rejected headers from older clients). That was quite simply a screwup on our part.

So we've had our share of screw-ups, no question. But lately I've been reading the "breaks at every release" meme more and more. Based on the evidence I have, it seems incredibly disconnected from reality.

What am I missing?

You're missing the fact that docker clients complain about the API version if they're different from the server - regardless of actual compatibility problems.

http://stackoverflow.com/questions/37617400/ sums it up. There's a magic (afaik undocumented) env var DOCKER_APIVERSION you can set for compatibility, but nobody can find it.

If you can't mix even minor versions then yes, it's a problem. I can't run X in prod and anything other than X in dev of I'm working with the same toolset.

The issue you are referring to happens only if the docker client version is newer than the docker daemon version. In the case where the client is older than the daemon, it has always worked.

Thanks for bringing this issue up though, because we are currently working on a fix. Here is the PR: https://github.com/docker/docker/pull/27745

As for DOCKER_API_VERSION, it is possibly not well documented, but it is indeed present in the documentation: https://docs.docker.com/engine/reference/commandline/cli/

Hopefully this was helpful.

The client will nearly always be newer, so that message will almost always show up. You'll update your laptop tools regularly, and only bump your infrastructure occasionally.

How many versions has this been a problem? It's not thinking about things like this until (apparently) 11 days ago that give Docker haters such ammo. The UX of the entire ecosystem feels like an afterthought. Yes, it's been evolving rapidly, but that's no excuse for not having a good, unified user experience for what's there or what's coming next.

That's a fair statement, especially with d4mac/d4win and auto-update.

Posting a pull request and thinking about a problem are two very different things. The project has chosen to err on the side of safety (ie, user asked for some feature to be used but it is silently ignored since it's connecting to an older daemon) in this regard. -- this is not the only problem to solve

Maintainer here,

After verifying, I have to rectify your second point: 1.12 DOES NOT BREAK older clients.

To add more information, the issue was present for Docker clients with version 1.9 and older talking to THEN Docker master. Thankfully it was seen before 1.11 was released, and thus we decided to stay on Go 1.5 for Docker 1.11 for this reason.

For 1.12, we added a workaround for this issue, allowing us to upgrade to Go 1.6.

UPDATE: Docker 1.10 was built with Go 1.5 so this issue was not present with a Docker 1.9 client talking to a 1.10 daemon either.

Oh, hai there. I haven't gone 'whole hog' on Docker yet and subsequently tried to push for rolling it out in production for my team.

I don't know how incorrect I am, but it seems like the Docker project is more interested in features rather than stability. Except, like you said, I don't have any specific examples to back that up. Maybe it's just community hive mind thought bleeding onto my decision making abilities?

The most concrete failing I've experienced was the deprecation of the boot2docker project in favor of docker machine. When I started using docker machine, stuff just... broke... and it left a bad taste in my mouth. In fact, using docker on either a mac or windows is just awful for me. My anecdotal experience is that it breaks in small ways that then that can really disrupt my productivity. I'm in the process of picking up docker again and it led me to install Arch on my work laptop just so I can have native containers.

Right now my use case is using containers to test Chef cookbooks and it works... alright. Not as well as virtual machines, but alright. How docker handles init bothers me a little bit but it's nothing the maintainers of the testing suites for Chef couldn't handle through yaml configuration options.

Oh, I guess while you're here... one thing I've been wrestling over is the usage of dockerfiles. I have no problem with containers but to me, it looks like using a configuration management system to bake container images seems to grant so much more flexibility and testing capabilities that I can't really see dockerfiles as anything other than an intermediate step. Is this a valid viewpoint?

"but it seems like the Docker project is more interested in features rather than stability."

I think that is a fair dig but I think they are still exploring the limits of what it can do. They are following the path of release often and make changes and show your working really well - you get to see the code and can contribute yourself.

The thing is that you have to take all that into account when you are using their product. You need to take an Engineering approach to "production". Use Docker in the way that works for you. You state that you ended up using Arch instead of Windows or Mac - so fine, you have constructed a tool chain that works for you and welcome aboard! You see, Windows and Mac also do odd things every now and then but you don't get to see the workings.

You are clearly not daft - you use Arch. I run Arch (my wife's laptop), Gentoo (my laptop and a fair few mail proxies and Icinga instances and other things), a lot of Ubuntus and a slack handful of Kalis and a sprinkling of Debians, Centos and others.

You need to use engineering principles when going to production. Also you have to look at the price you paid to use Docker - it's a purely social one but one that is extremely important and may not involve money but cost you a fortune anyway. You need to decide if you want to pay that cost.

You are clearly someone who understands the open source methodology but I would suggest that you look a little deeper into the dev -> prod thing. There is no reason why dev = Docker => prod = Docker. That's something that only you can decide after careful testing.

Cheers Jon

Thanks for stopping by. I too would like to see something more than a "my toys broke - mwaaaah".

I personally find Docker a phenomenal resource for development. For production however, you analyze your use case, the risks, the failure modes etc etc and then you decide whether Docker is a part of that.

I'm by no means a massive Docker user but compared to many of the sub systems running on my laptop it has been very stable for me although I do run Gentoo. Actually my entire laptop is surprisingly stable TBH. I absolutely love the way I run a few short commands and a fully tooled up web app with all the supporting infrastructure simply appears.

Funnily enough, only yesterday I had a look at this: https://nextcloud.com/collaboraonline/ which I found in an article somewhere (LWN?) Yep - love it (NC) so the next step will be to decide how to use it for my home production system (yes I have dev/production at home) For that I will use pretty much the same criteria that I will use for my office and finally my customer systems. In this case Docker for production probably makes sense but I need to analyze it a lot more. A CoreOS VM will probably be spun up very soon in my attic for stage 2 in home production.

I really don't like to see rants along the lines of the other article, denigrating a product that a lot of people have worked hard on, that shows a lack of discipline and clarity of thought. I for one would like to thank you for your work and making it available for all. It has genuinely saved me time and effort and made my IT life a little bit better.

To answer your final question: a thicker skin, mate 8)

Cheers Jon

Hi There,

I created an account here so I could let you know I've clarified some points in my post to address your concerns.

> Has anyone here encountered an actual breaking change?

The fact that if your local client is newer than the daemon, the client will fail to do anything is pretty annoying. You can hack around it, but that's not a solution to the problem.

Also, while you shake off the 1.10 change as though it wasn't a huge deal, the migration tool that was written was not fit for purpose. First of all, it sometimes caused corruption after the migration (fixed after release but still an issue, and you couldn't practically un-bork the corruption). There were other issues with the tool, but the biggest issue was the there wasn't any sane transition period -- which made packaging very difficult. Because Docker couldn't handle restarting the daemon and keeping the containers around (and after restarting a post-1.10 daemon you'd have to wait for the full migration to finish before you could even interact with the daemon -- another issue with the transition), you'd have to publish a fake version bump of the Docker package to even alert users to the issue!

A more sane way to handle it would've been how databases handle the issue. You lazily do the conversion (or you have your software be able to handle both formats so you can run your migration while still having your containers work). And yes, I'm aware that it'd be difficult to implement but the transition path implemented was really bad.

I spent several weeks of my life (that I'll never get back) trying to make the migration path work. It didn't end up working because it wasn't designed to work nicely from the outset. So I ended up just telling users of our Docker packages about the issue and packaging the migration tool separately.

Another issue that I've had recently is that Docker is making it increasingly difficult to use `runc` by another name. In openSUSE we package a system runC separately to Docker (and it lives in /usr/sbin/runc), but because of the fact that in 1.12 --runtime was intentionally broken (and containerd's defaults are no longer obeyed by Docker) now you can't effectively use --exec-opt native.cgroupdriver=systemd. Great. So now we're going to have to package docker-runc and docker-containerd separately.

Don't get me started on the containerd split. I was not very happy the day that PR was merged, because it caused a bunch of headaches for us. It's also a bit amusing to note that containerd was tagged as "pre-alpha" until Docker bundled it (then it was considered stable).

That all being said, Docker is a great project -- it's just that you have to admit that issues with Docker are not "incredibly disconnected from reality".

This seems less of a "retort" and more of a validation that most of the issues brought up in the original article are valid complaints.

I love how the major issue, that both this article, and the original article warn about is: don't use docker on 'CORE APPS'....

That says all you need to know about the trustworthiness of Docker.

EVEN DOCKER PROPONENTS caution against using it in 'important' apps....

What apps are people investing time in that aren't 'important'?

Is there a coffee machine that is ok to use for a docker app somewhere?

> Is there a coffee machine that is ok to use for a docker app somewhere?

Most coffee machines are docker ready.

To guarantee you the best experience, you will need to setup a pair of coffee machines, plus an orchestration system that will be responsible for swapping them automatically when one ran out of coffee.

Note: There are only prototype of orchestration systems. Nothing for sale in the corner shop yet.

---

More seriously...

Not important: Most internal, development, and test systems

Somewhat important: Web applications, various support micro services. (They all are stateless, with multiple instances, and reactive failover by their respective load balancers).

Critical: Most databases (especially the ones without multi-master mode and automatic failover), trading applications, payment systems, accounting systems, databases with money $$$

Haha a coffee load balancer. I love that idea. I'm sure Amazon will have a coffee machine web service they offer for $2000/month. ACS.

This whole ordeal has actually been a really really good lesson for me about mature vs less than mature apps. It's been really enlightening to read the posts and discussion.

AWS was also considered not a fit for running "core" apps and databases... look where we are now.

Docker is certainly having some teething issues - and things like kubernetes are certainly WIP. But I wouldn't be as skeptical - plenty of companies use it in production already, for "core" apps.

Interesting, yeah I truly see the potential of the idea. It's definitely the future. Really glad to see Google and others getting on board and giving them the resources they need.

I interpret that as don't use X (in this case Docker) with your most essential services. Initially. Get your teeth cut, make some mistakes, establish routines, etc with your internal apps, your low traffic services, etc. Be patient, then once comfortable and confident your start to roll it out to your core, heavily integrated services in production.

Having just come off a year at large(very large) government department that have 100s of core services running in Docker in production, I think the tech is ready, but not sure if every company is ready.

Hmmmm, if you need ton's of workarounds and there's a huge learning curve to learn the weird quirks(bugs), I don't think the tech is necessarily stable...

> Again, well accepted principle that “thou shalt not run a database inside a container”. Don’t do it, end of story.

Sorry, but this is really a bad advice. We have ran and contine to run various databases inside Docker including MySQL, PostgRedis, Cassandra, Elastic Search, RethinkDB even HDFS with proper user rights and configuration. We can maintain the state just as fine. If your only problem is to move the data, all you have to do is stop, export, tar it, move to another server, just as you would do in a normal server. Docker is not a magic bullet to solve such kind of issues. Yes, Docker might have another problems, but just as you could not run someting with state inside Docker does not mean "thou shalt not run" , there are various ways to manage state. Host, IO can get crash regardless of Docker.

What problem does containers solve for you for this particular part of your infrastructure? Native storage software packages are available for mainstream OSes that handle dependencies via the native package manager. And since the storage they manage is usually directly attached, nodes that run this software are infrequently migrated. And this software is infrequently upgraded under the maxim "if it ain't broke, don't fix it." It's there to store data on behalf of the applications you write; it is not a thing to upgrade or migrate for its own sake unless there is a bug to quash or a new feature that your application will depend upon; and even then, migrations must be carefully planned to preserve availability for users.

Docker and the like seem like a solution in search of a problem for this particular part of a typical service infrastructure.

Or, to put it more bluntly, just because you've gotten away with it (thus far) doesn't make it a good idea.

> Or, to put it more bluntly, just because you've gotten away with it (thus far) doesn't make it a good idea.

What is your typical server infrastructure? We use bare machines and had kernel panics very rarely (only with experimental MACVLAN). We did not 'get away' with anything. We have been using Docker for over 2 years, and LXC much more and know how to handle mounted directories, it is not a rocket science. We used LXC, now Docker to provide quick isolated mini linux environments. They are just different type of Virtual Machines when you abstract it in your head, and do not expect it to do magical stuff like saving your data and send it to another host upon crash.

> Native storage software packages are available for mainstream OSes that handle dependencies via the native package manager.

They are often outdated, we mostly get one of the latest builts, and in some cases Snapshot builds, where they are not available anywhere. So ability to run versioned instances of some software independently and isolated is a huge win for us. We do not want to distrub people using the old version. Yes it can be solved with VMs. We use our own bare metal servers. We do not like to "pay" to open VMs, so we turned to OpenStack. It is very complicated for our use case, we like to use bare metal on our own. But, we still need isolation, this is where containers; LXC and Docker came to help and saved us a burden.

> They are often outdated, we mostly get one of the latest builts, and in some cases Snapshot builds,

That's a very usual thing to say about databases. For critical and commercially supported things such as your relational databases, nobody runs snapshot builds. You run whatever builds your vendor supports, and for all those you mentioned (and most other vendors) that is going to be native packages.

Native packages are nothing more than packaged software with startup and maintaining scripts, (and tested) so as long as you can manage these, I really don't see I have to stick to what my vendor thinks best.

I can't imagine what your use case is that requires running bleeding edge database servers in production and that cannot be managed via native OS packaging. Would you please enlighten us?

Otherwise it really sounds like you're using a non-best practice (running a DB in a container) to paper over another non-best practice (treating your DB as a non-sacrosanct piece of infrastructure by introducing intentional instability through frequent needless upgrades).

I really don't understand why you insist. You keep saying "best-practice" like it is a law, but the truth is there is no such thing is best. It fits our cases, worked well, continues to work well. Did I say it cannot be managed with traditional package managers? No. But I do not like upstart nor systemd, and I like the simplicty I get from Docker, and ability to replicate the almost same environment easily. I really don't care a guy on the internet that is upset because I don't follow their "best" practice. I gave you my use cases, said that Its workinng well for almost 2 years and list the advantages, you still focus on your "best" cases. It works just fine as long as you know what you are doing, and I expect to hear more justifiable and technical explanations why this should not be done, rather than your feelings and biased best practices that does not translate into any verifiable points.

Well, for one, because many database vendors won't provide support unless you're using the bits they've precompiled for you. If you think you'll never need outside help for the lifetime of your organization, then more power to you.

Another reason is that you're introducing an unnecessary dependency/component that can introduce potential instability. Common sense (since you seem to be offended by the term "best practice") and experience suggests that you should eliminate as many sources of potential instability as possible to increase the overall availability of a service or component.

And this is exactly why you should containerise your db's run environment with docker. Dependency management. You cannot control your host environment to the same degree you can control the docker environment, given the same amount of effort. If Oracle only supports RH6 and you use Ubuntu LTS on your host, you run it in a RH6 based container. Oh look, maybe even use the official one https://github.com/oracle/docker-images

If the DB vendor is supplying and supporting the images, then that's a different story, of course. The devil is in the details, and the presence of Docker support shouldn't be interpreted to suggest that other aspects of database administration shouldn't be followed.

At any rate, at least MySQL and its Percona variant have native packages for modern mainstream Linux distributions , so as a practical matter you shouldn't need to run them in a container: they've already worked out the dependencies for you. This means you don't need to bother with the overhead of Docker.

I update my database servers for the same reason I update my proxy servers: more features, security fixes, better performance, increased stability.

How frequently? And can you quantify those benefits relative to the risk and amount of work involved? How do you know it passes the cost/benefit ratio test?

The same as all my other software?

Is this your first time working with it?

You didn't answer any of the questions expect the one about frequency. I'm left to assume that your usage is so irrelevant or your user base so small that you haven't needed to spend time thinking really hard about operational difficulties, SLAs, or calculating risk tolerance relative to value. Or gotten yourself into trouble yet which every DBA/SRE of a database of substantial size gets into eventually.

I didn't say exactly how frequently, just that it's the same with my database as other software.

That's... not helpful at all.

I'm still not sure I follow. If you're not setting up a DB that will have such heavy usage that it needs bare metal access, what's wrong with setting it up such that it stores data on a NAS, or via mounted volumes? Then when it comes time to upgrade due to a security issue, you can create a new container, spin it up, and if it barfs, relaunch the old one.

Unnecssesary complexity. A VM would've sufficed for what you described.

What is the "Unnecssesary complexity" in this scenario? You literally just run a command and you can spin up a new container with your data.

I find VMs are super slow for development purposes (I mean setup, starting up and shutting down rather than actual processing), where docker's made a massive difference. YMMV especially if you're not spinning up new projects frequently.

They might be a better solution in production, but I have a strong preference for production mimicking development.

Precisely the point. It's fine in dev but once you are running in prod and need rock solid stability the generally accepted dogma is that containerizing your DB is playing with fire. As you said, YMMV.

A good reason for running a database in a container is fault tolerance without setting up database replication.

Put your database in a container, let it store its data on a network storage system (like a distributed file system). The container scheduler makes sure it gets rescheduled when the node goes down.

And then the NFS handle goes stale, and the fun begins.

And what does it do any better than doing the same (keeping database on an NFS/iSCSI/AoE share) on physical or virtual machines?

Less downtime and automatic recovery, I'd say. When the host machine goes down, the scheduler will pretty quickly start the container on a new machine. There the container will startup much quicker than a VM would.

> Less downtime and automatic recovery, I'd say.

How? By magic dust sprinkled by Docker Inc?

> When the host machine goes down, the scheduler will pretty quickly start the container on a new machine. There the container will startup much quicker than a VM would.

You seem to forget that a) container's filesystem needs to be downloaded in the first place, and b) that VMs can work in a standby mode, not running the service (database) when not needed. Starting a container seems to be much more fragile solution.

There are already well-developed mechanisms for things like that, and we (sysadmins) were using them for dozens years already.

The question still stands: what do containers do better than the mechanisms we had for a long time?

Its faster virtual machines, which do not have live migration for that specific use case. Just because you are doing it for years does not mean that it cannot be improved or have an alternative with its both downsides and upsides.

You do not want a 'downside' with any database. Especially when big money are involved, people are shown the door for far lesser issues than the ones that docker seems to have.

By the way, you still haven't answered his question.

You're praising a feature that is an anti-pattern for database administration. If you're migrating data around and upgrading and restarting your production database without adequate business justification, you're asking for trouble.

The HN discussion of the article being retorted: https://news.ycombinator.com/item?id=12872304

Re ECR (the EC2 Container Registry), it has one downside that the author doesn't mention, which also applies to Google's own registry.

A Docker registry has own authentication system. So does AWS (and GCloud). So what you end up is one wrapping the other: To access the ECR, you have to run an AWS command to get a token to put into "docker login". Google has "gcloud docker login" for the same purposes. Both produce temporary credentials that time out, so can't be used for long-running things.

This means that any tool designed to work with a Docker registry needs to support this particular workflow. For example, this affects Drone [1].

It also adds complexity. GCloud is particularly heavy on the authentication complexity side already (compared to AWS's comparatively simple keypair approach), and with SSH, GCR and Kubernetes on top it starts to stack up in ways which can make users' head spin.

Straight Docker Hub is refreshingly straightforward by comparison.

[1] http://readme.drone.io (not to be confused with drone.io)

At least with Google's GCR, this is no longer the case. They have a Docker credentials helper [1] which transparently handles authentication for regular interactive use, and service accounts with permanent keys [2] for use with CI servers etc. I find both of these to be fairly straightforward to use.

[1] https://github.com/GoogleCloudPlatform/docker-credential-gcr

[2] https://cloud.google.com/container-registry/docs/advanced-au...

Previous article: "This is nowhere near ready for those who just want to get the job done."

This article: "It'll be better in the future, you'll see!"

The former is verifiable, the latter is a hypothesis.

No what I was saying was its ready for today in certain use cases. But only if you look at it as a long term transition and not a short term project

As an example of Docker in production: Expedia are moving lots of their legacy infrastructure into Docker containers. My third-party contracting team that works on projects for Expedia (we're brought in so the rules and bureaucracy don't apply to use, allowing us to rapidly iterate and experiment in ways the core teams can't) have been using Docker end-to-end (local development through to autoscaled production deploys)

While there were teething issues, this article does a good job of pointing out the flaws in the original article, I think. It's been easier to get our team up to speed on Docker and it's gotchas than nearly any other configuration management, server management, et al. systems that we tried!

Short version: Nope :p

Long version: I met the DevOps guy who [I believe] is responsible for pushing Docker at Expedia and we've had long conversations about it.

They were lucky to have had a particular environment and a specific version that worked, and got it pinned down and frozen very early.

I suppose you are on the dev side and not aware of all that. (Hell, maybe, you're not even in the same subsidiary of Expedia). I'm glad it all worked out for you as a dev, my devs are also happy with Docker. (We're probably used as an example of Docker success story at times).

In the end, there is no free lunch. There is dirty work done and more to be done. Some of which is invisible.

Sorry if I didn't explain it correctly!

We're an external team (completely seperate to Expedia) that is brought in to build certain products :)

And yeah, definitely. I spent a couple weeks debugging the ECS issues we had with the Wotif guys, which was fun, but none of the issues were insurmountable!

We were one of the first adopters for the new ECS deployment, and while some parts of it weren't fun (Splunk still drives me mental) for the most part it went smoothly: that's down to how good the ops team is, I think.

Our specific case was somewhat special, in that because we're outside of the rest of the infrastructure, and had extensive experience with Docker in production for other clients, our apps were basically ready to rock from the get-go. I think we were the first PHP deployment within the new ECS deployments!

Should send me an email, it's in my profile, would love to chat sometime!

Can anyone comment on how rkt compares to Docker regarding the issues from this article ? And how does rkt compare to Docker in production in your experience ?

I've been using Docker in production for a single server website and have had very few issues. I do like how easy it is to reproduce a working environnement with a "docker build" though.

That being said, I think that just using Ansible on a server is probably an easier and more reliable solution. Ansible is battle tested and allows to have reproducible environments too.

Is ansible still python 2.7 only?

Yes, and it will still stay so until RHEL 5 is EOL (next year I think), so the Python 2.4 support requirement can be dropped. This is the main showstopper.

There is also work in progress:

https://docs.ansible.com/ansible/python_3_support.html

They've been working on it, but it might just be the control machine that's 2/3 ready.

Related question, what happens when a docker images gets pop'd.... how do you keep it around for investigation, does it get imaged for later forensics? Every time I have asked people in IRL doing docker, they seem to focus on updating/patch... on how easy that is and moving on... but that is not always an option for every client. Do you just image all docker images before they get terminated/migrated?

All the containers are available after termination, yes, so you can investigate.

> So the point is valid, but there are some big names invested in solving it, so I’m optimistic we’ll see some stability in the future

And it will still be valid if someone forks Docker. In fact, that would validate the criticism.

Are these breaking changes problems caused by Docker itself? I was contacted by Docker and was considering applying, but it sounds like their engineering management doesn't know what they're doing. Is this depiction accurate or is it overblown?

I would recommend investigating the matter yourself, and making your own opinion. What specifically has been broken in past Docker releases? How have they handled it? How would you have handled it? If you do decide to interview with Docker, make sure to bring up your findings, especially areas where you think they screwed up. This has two advantages: you can see how they react to constructive criticism, and they can observe that you are capable of making your own opinion and providing constructive criticism.

If you really want to impress your interviewers, back up your criticism with a pull requests fixing the issue. That will automatically put you on top of the pile of resumes.

Anyone know why Erlang doesn't run well on containerized Docker?

Hello. It was a long time ago and I didn't get all the details.

I think it either came down to a broken version of docker at the time that couldn't run properly.

Or it was because of unstability from Docker. All our Erlang apps are meant to take thousands of requests per second and Erlang have particular ideas about how the system/network should behave. It's very hard on Docker and any unstability or incompatibly would make up for outstanding failures.

Either way, we know for sure now that there were many major critical issues with Docker at the time of the Erlang trial. Not adopting it was the right decision, and it still is.

We only use Erlang for critical applications. We really can't afford to push the limits after a few red flags.

I have used Erlang and containers, but not in combination. I know that Erlang (really, beam) has some ideas about what the network should behave like, and highly virtualized/"software defined " networking may confuse it.

What issues are you having? Containers are not really different from non containerised environments. Have you filed an issue?

It's something I read in another HN thread

It runs fine. However, the challenges are the same as any other environment (e.g. AWS): Remember to set a Erlang cookie, and make sure the host name is sane/set and resolvable, otherwise you will run into issues with the portmapper (epmd).

I run several erlang/elixir applications in containers, it works just fine. I'm not sure why someone claimed it doesn't, but I suspect they didn't try very hard if they did have trouble.

Chokes under significant load, perhaps? Do your erl/iex apps receive much traffic?

My apps are handling events from devices, a great many of them, probably ~1k/sec. That said, it could certainly depend on the app.

Why not use an erlang unikernal instead?

if I am using mount Volumens to export my data, can I bypass the aufs/overlay implementation/logic ? do I need to pay attention only if I don't mount the volumes? thx

Volume mounts do not use the aufs/pverlay drivers, no. These are used for build, and for constructing the rott filesystem for running containers so files are shared.

Docker seems very limited when it's unsuitable to run databases.

I've never seen this limitation with other container solutions. What is it about Docker that makes it problematic?

Nothing. It's bad advice. Lots of companies run databases in containers.

Most likely lack of planning and applying a it should "just work" process to the scenario at hand...

The thing I've learnt with Docker is that all the other Prod issues still exist. Docker at its core only solves the executable part or distribution of the process to run. Still need to figure out, network, storage, monitoring, backups, discovery, etc...

What starts out as a single host can quickly quadruple once all the other considerations are taken into account and wanting a scalable, reliable and available system.

Good retort. The original article seemed clueless, the part about aufs was just wrong, the complains about the apt repo exaggerated. Running docker on Debian ancient is kind of brave, though. And software is finished after five years, maybe in the financial industry. Currently development has such a pace, I'd say after five years it's abandoned and replaced.

"The internet has been a wash with a well written article about ..."

Typo: a wash => awash

I know! The content is more important than the quality of the writing, but it's a little surprising to see such a mistake jumping at the reader at the start of an article. We should go back to the first days of the Internet where "updates" were possible. :) I would have loved to suggest an update quickly.

My googling failed on this account on the correct grammatical term.

Reply on news.ycombinator.com