Why Databases Are Not for Docker Containers

This article is terrible. It's a lot of wishy-washy explanations devoid of technical detail - because their isn't a technical explanation or justification for this list.

I've run extensive benchmarks of Hadoop/HBase in Docker containers, and there is no performance difference. There is no stability difference (oh a node might crash? Welcome to thing which happens every day across a 300 machine cluster).

Any clustered database setup should recover from failed nodes. Any regular relational database should be pretty close to automated failover with replicated backups and an alert email. Containerization doesn't make this better or worse, but it helps a lot with testing and deployment.

> But what about Configuration Management systems? They’re designed to solve this kind of routine by running one command.

The problem with this for most of the developers you see praising containers, is that with a containerized setup, you've already got the rest of your deployment process down to `docker service update --image myorg/myservice:1.3.0 myservice`

(And, in fact, maybe you're even running that code against immutable-infrastructure container-host OS like CoreOS.)

And now, you're suggesting that these developers would have to add this whole other process just for managing the deployment of the DBMS package—and probably the OS it's running on, too. (Maybe they would even have to add process to manage the VM it's running on, if they were doing everything else until now using autoscaling + swarm auto-join.)

Developers put DBMSes in containers because they're developers, not DBAs. If you are a DBA, then obviously this will seem wrong to you. A DBA wants to manage a DBMS using the DBMS's tooling. Developers, meanwhile, essentially want to manage DBMSes as part of the same "release" as their apps—being able to "pin a dependency" to a specific DBMS version; update the DBMS as part of a commit and see the whole updated stack go through CI to integration-test it; etc. These are development-time concerns that—at small scale—usually override operation-time concerns.

Alright, all you brave people who run databases in docker: where do you store the actual data?

- Host mounts? So what happens when a container gets rescheduled to another node?

- Docker volumes? What happens when a container gets rescheduled to another node?

- External SAN? Congratulations on your budget. That's not easily doable in public cloud I guess?

- Shared filesystem like NFS, or Ceph? How's the performance for you? And the filesystem itself runs outside of your docker cloud I guess?

- In the containers themselves? So you probably run some clustered database. So what if disaster strikes and all your containers go down? And get moved around?

Also, databases in many cases need to be tuned for performance. How do you do that in a cloud?

Maybe most of you are not running container scheduling. Which really is only taking containers half way.

I am so surprised and disappointed that such a shallow article has made it to the top. It provides absolutely no value.

Most of the upvotes (gathering a consensus from comments) are not because they believe that Docker is not the right tool but because they have been frustrated by the ops part of things.

I'm also surprised at how many think that one tool will come and solve all their problems. Guys, it doesn't work that way. Docker had one job and it does it fairly well - process isolation for humans (ok the engine goes crazy sometimes but hey everything does). For all the other things, you need to setup your own workflows, tools and processes.

Tomorrow, another rant article would come at how apt sucks but in fact, apt and friends are just an amazing example of how packaging should be done. It is not ideal, but it works! Everything fails sometimes and that is when new things for it come up.

If every immature developer started adding rants on the internet, we would pretty much be disregarding half of the software. To the OP, if you are not able to achieve something, please don't rant just because you couldn't do it.

Part of the problem is that you're using databases that can't cope with failure. In large scale production systems things fail all the time. If you've got tech that can cope with failure it's not an issue.

Additionally, Docker is pretty handy when you're attempting to manage clusters consisting of thousands of nodes. In that instance enforcing best practices, automating workflows, scaling teams, auditing and preventing configuration drift are much bigger problems than a single server failing.

As others have mentioned, Docker doesn't really do any magic which might harm the smooth running of a database, but just leverages process isolation built into the Linux kernel and provides a convenient way to package and distribute bundles of software. In the case of a database, the former can be handy any time you want to run a database on a host where you want to run other software too. As for the latter, being able to run the same configuration locally as on production servers, to replicate configuration over many nodes in a cluster, to distil both configuration and software into atomic units are as advantageous for databases as for any other software.

At my company we have been using PostgreSQL on Docker for over two years without and have been sufficiently satisfied with the results that we're in the process of turning out setup into a product in its own right: http://containable.co/

Kubernetes StatefulSets [1] are intended to address this kind of use case. They provide stable network identity and stable storage.

Kubernetes enables a container to declare the resources that it requires, including things like dedicated CPU and memory requirements. There are still some rough edges (example: how do you set the amount of kernel shared memory you need), but those issues are being ironed out.

[1] https://kubernetes.io/docs/tutorials/stateful-application/ba...

I've been using docker in production with Elasticsearch and MySQL for 3 years in the PB scale and have never had data corruption issues occur.

Corruption occurs on data drives even without docker - you still have to plan for it. This is why you enable replication. This is why you snapshot/backup your data daily and have disaster recovery plans.

There are some major reasons why I actually think running databases in docker containers, even if you are mounting a volume for the data.

1) Development environments can be similar to production. Ensures everyone runs the same version that is running in prod.

2) You don't have to worry as much about what is installed on the host machine.

3) In a clustered setup, it's easier to ensure each node is running the same configuration, version, etc...

One of my issues with all the gripes about docker are the assertions that it causes issues. In all of my time of using docker, 99% of the time when there is an issue it has nothing to do with docker itself. Everyone loves to blame it when things go wrong though.

This article doesn't really back up any of the claims about any of its issues. It just makes blanket statements without backing them up. Don't like docker's networking? Use host networking then.

What people don't think about is the countless issues that will never come up when using containerization. I never have to worry about whether or not python 2.7 is installed on a server that I'm going to deploy a python 3 app on. I also have MUCH higher confidence that if things work on my local development env (which runs the same containers), then there is a high chance it will work in production.

YMMV

In the ancient times of databases we had to make sure that we wrote to 'raw' disk devices bypassing any possibility of the operating system file caching/buffering failing to 'flush' our writes all the way to the disk. There was an implied guarantee that what we thought we wrote was actually written to disk.

In today's world of 'virtual' everything which may sometimes be many levels removed from the raw disk devices, how do we still ensure that a write to a database is still a write to a physical device as opposed to an incomplete write that looks like a completed write to a higher level virtual disk? Is there a guarantee that everything is flushed to a physical disk?

I still can't really figure out what a container is. Every time I think of a use case for one, I read something like this which says that's a terrible idea.

The use-case I need solved most often is the following:

Create a standalone "server" that accepts and responds to network traffic, has some way to store data, and whose dependencies (i.e. system packages, frameworks, etc) I can manage independently of any of the other "servers" I have running. Do I just want a bunch of VMs? Or docker instances that all point to some other DB (that's apparently not in a docker instance...?). But then they're no longer independent from one another because they all use the same DB. So do I need a separate DB for each serverlet? Which lives where? On its own VM?

This article mentions offhand that the storage drivers are unreliable, even for data volumes.

Is that actually the case? Is there a serious risk that a database will be corrupted by a container crash, as the article claims? A regular crash of the computer should not be able to corrupt a database, is a container more dangerous in this regard?

The basic problem is that a database server is not what Docker is trying to containerize. They want to containerize applications which used to be clear cut things. They used to be single purpose things built by compiling and linking some source code into a single binary. They did not contain everything including the kitchen sink and a plumbing kit complete with automatic drain clearing snake.

Today's database server, and this includes more than just the RDBMSes, are actually a collection of applications. Even if the developers have the bright idea of integrating it all into one binary, it is still not a traditional app. It is a collection of apps built into one binary, like busybox.

To truly dockerize a db server, it would need to be built differently, as a collection of separate, semi-independent apps, that have clearly defined interfaces between each other. Until that day arrives better to use something like Ansible to manage your monolithic db server on it's own instance.

Docker doesn't buy you anything with db servers because you generally want stability. I know some people are experimenting with highly scalable clusters of db servers, and using things like MySQL in a way that was never intended, but they know they are on the bleeding edge. I also expect them to soon start hacking away the bits of the RDBMS monolith that they do not need, and building single purpose cluster members that only do one of the jobs in a normal RDBMS. It might work; give them time.

But if you have to run a db to support your business, don't do it with Docker. I run PostgreSQL and right beside the monolithic RDBMS there is a docker host that runs miscellaneous support stuff like serving up pgBadger data, running a REST interface to data, running an app that listens to PostgreSQL NOTIFY events using Camel pgconnect, and some other admin tools (simple webapps to do db related stuff). Docker has a role, but running the main RDBMS is not it.

"You may corrupt the data in case of container crash where database didn’t shutdown correctly. And lose the most important part of your service."

This is the point in the article where you know you don't need to read the rest. If your data integrity hinges on your database being able to shut down correctly you will be disappointed.

The author believes in fairytales.

Docker is a packaging tool. How you chose to deploy your software package has nothing to do with your operational and DB administration procedures. Mixing the two topics is very confusing.

Hi everyone. I'm this immature who's made this blog post. Thanks for the feedback, I didn't expect so much people to care about my post, cause I mostly lead the blog for myself. There were about 20000 users seeing this while I had only 3000 for the last year. So, if you have any questions for me as an author - please, write them down at this thread. I'll try to deal with all comments soon.

All system can fail, and failed system should be recovered. That's why we have disaster recovery plan, back up or replication etc.

(I have just shallow knowledge in docker volume, so please reply if not corrects exist) I understand docker with local volume is just abstracted file system have mounted path that volume specified linked with host path. So file i/o is probably not a problem in local volume.

And if network has bug that make data corruption can make difference between nodes, docker cannot be used any system. So we can think network bug may not make data corruption(but can make network separation).

Now I am building on-premise autometic deploying software using Kubernetes as a outsourcing job, so I tried to find SAN for resolving stateful data. After many searching, I realize only local path will guarantee stability of database filesystem. So we mark storage node, and all type of stateful app(limited kinds by playform) is deploied on that node. So we can easily back up and manage storages.

As a deployment manager and backup automation, container for database have a great functionality. All file produced by container are jailed where I specified and can be copied or backed up. (for stabaility, replication is first class. pause-backup-resume or copying on running will make operational unstability. you can use both for backup, replication first and make backup using that replication node)

We have been running stateful services including DBs inside of Kubernetes for a while with some success. Outside of a few docker bugs it's definitely been worth it: https://medium.com/the-smyte-blog/counting-with-domain-speci...

Its like i'm reading blogs about running databases in vmware all over again, just 10 years later...

There is actually little difference between a process running in a container and in the host. They are using the same network and ideally the same filesystem so there should be no difference whether you run the database or app in the host or the container.

Every container article on HN seems to perpetuate more confusion on containers and often arbitrary misguded rules on what a container should be confusing new users even more.

The problem is Docker has taken fundamental technologies developed by other people and wrapped it and since they do seem to want to give credit and pretend to be more than what they are they obfuscate things in words like docker filesystem drivers, networking drivers, container drivers etc. Untill users get familiar with the underlying technologies this sorry state of affairs will persist.

A container is simply a process launched in its own namespace thanks to kernel namespaces introduced in Linux 2.6. Its got nothing to do with cgroups, cgroups 'can' be used to limit resources to container process by cpu, memory or network resources if you want. If you launch this process by chrooting (or pivot root) into a basic linux rootfs filesystem you have a container. If you launch an init in this process you have a LXC container. If you don't and prefer to launch the process directly from the host you have the Docker version of the LXC container which is a fussy hack as now your container is not contained and runs app process not designed to be run in pid 1 in pid 1 and needs to be managed from the host. Kudos. You can also add a network namespace to the container process so its has its own network layer.

The biggest problem currently is a lot of Linux subsystems are not namespace aware and you can't really do proper isolation. Even cgroups only recently got namespace support. Anyone know who these folks are who are doing all this fundamental work?

The second biggest problem is layers are oversold, their actual practical use is marginal at best. They are also complicated and buggy with multiple issues with running overlayfs or aufs on xfs, databases and btrfs. The third biggest problem is a lot of projects and teams working on Linux containers are pushed into the background or marginalized and misrepresented like LXC was by Docker devs instead of giving proper credit and explanations.

The talented developers of overlayfs, aufs for instance are virtual unknowns in the container ecosystem inspite of Docker fundamentally dependent on them. These guys can solve a a lot of the problems with containers but first users must know about them and support them so that bugs can be fixed, rather than have the Docker team create more workarounds and hacks.

If you limit your understanding of "containers" by not advancing past single-page tutorials produced by content marketing folks at orchestration startups, you may _feel_ like the author is right. But as it almost always the case with damn computers and generalized topics, there's no right or wrong. The world is boring and full of "it depends" but that was conveniently left out of the article because the goal, I suspect, was to back a sensationalist title and produce clicks/views. But I'll bite:

> 1. Data insecurity

The author is mixing up Docker image store with database's own data. It is true that Docker graph drivers have issues, but they don't store any data, those are binaries you distribute and you're welcome to start docker containers from a plain old directory on disk. Layers are sexy but optional and they have nothing to do with your database data.

> 2. Specific resource requirements

The author talks about running additional processes on a database machine. Why is this an argument against containers? Maybe because containers make it somehow easier? I dunno... Yeah, don't overload your database servers with other stuff, containers aren't forcing you to do it.

> 3. Network problems

This one is the most bizarre, with statements from all over the map, basically saying "networks are hard". Riding unicycles is also hard, but that's not used as an argument against containers. Here's an obvious conclusion: if you don't feel like learning software-defined networks (or don't need the benefits they provide), then don't use them and run containers with native host networking.

> 4. State in computing environment

This port is just rambling, I do not see anything specific to reply to. If the point to make was that containers don't play nice with state, it's like saying "processes do not play nice with state" because that's what a container is: a Linux process. You have full control over where (pin it to DB machines only) and how it runs, use features you need (and understand) and don't use others.

> 5. They just don’t fit major Docker features

In this part the author is basically saying that it's easy (or easier) to install a database using configuration management tools instead of using something like Docker. True, there is more than one way to skin a cat and frankly you can use both a configuration management system and the containers. I just can't see how this can be used as an argument AGAINST anything.

> 6. Extra isolation is critical at the database layer

The author again claims the containers bring in significant overhead. That's simply not true. I would recommend to mentally replacing "container" with "process" when you read the orchestration blogs to see right through FUD. Again, you can run a container from a directory on your filesystem using host networking and it will be no different from any other process on the box. Using a network namespace does not add any measurable difference to performance. [1]

> 7. Cloud platform incompatibility

The title doesn't match the paragraph of the text that follows. The author basically claims that being provider-agnostic (one of the benefits of containers) is not valuable. Well, he's a database administrator and it's not valuable to _him_. But there's a huge business value of being able to run on different infrastructures: selling $100/mo SaaS subscriptions is nice, but when the stream of early adopters dries up and you set your aim at those nice six-figure enterprise license contracts, you may find out that you will need to be able to run on a VMware cluster in a corporate colo. And containers can help.

Containers are big not because they make developers happy, they're big because they let sophisticated companies significantly consolidate their workloads (via dynamic scheduling) and shrink their infrastructure footprints. I constantly get shocked by AWS bills people share with me and something like Kubernetes provide quite significant material value of shrinking them. But another less obvious advantage is the ability to run [1] the same SaaS stack on public and private infrastructure, opening up entirely new markets for your company. What's your revenue from China? Ever thought about containers being the perfect tool to penetrate The Firewall and run on your Chinese customer's servers? Anyway, those are good reasons to finally learn and use containers. And the reason not to? Well, not this blog post.

[1] We are https://gravitational.com and some of our customers ARE database vendors, happily running their mission critical (everyone is mission critical in our biz) workloads on containers / Kubernetes and deploying them into behind-firewall corporate clouds. So yes I am biased but I'm also qualified to respond.

Since you mentioned union fs then you are not using upstream kernel, most likely you are blaming your non-enterprise distribution choices on containers technology.

Fact #1 redhat do have enterprise docker container based solutions. Check project atomic and openshift

Fact #2 cloud providers like google, azure and amazon do have container basef solutions

Fact #3 coreos do have production grade docker based solutions

Fact #4 kubernetes do support pet pods aka stateful pods and can get data volume from reliable ebs or ceph

Upgrading database versions is mentioned briefly in the article, but is pretty much a show stopper at the moment for some databases.

e.g. Postgres (https://github.com/docker-library/postgres/issues/37)

Docker is an app store. Once you reach that point of understanding everything gets easier. On your Windows box or Ubuntu box you can just install whatever you want. Docker is more like IOS or android. When was the last time you edited an .ini file on android? It works, it doesn't, take it or leave it. Or you can clear the local data or reinstall. Not much else. Docker is the same way, at least by intent. No wonder they don't want you to store data on it, if you have a SQL database running on android how much would you expect out of it? Would you really expect it to be persistent? It's easy to install, that's the point, but it takes away a lot of freedom just like the app store(s).

Scaling everything else (stateless non-persistent services) is nearly trivial, with or without Docker. It's scaling databases where things get interesting, but here we are back to dark ages of DBAs and manual deployment :(

Of course the counter example of this is Google, which, as I understand it, runs everything on containers. It seems like if its good enough for Google, then its good enough for the rest of us.

I make an exception to this.. that would be something like Redis, or memcached as a localized (or perhaps sharded) cache cluster serving the systems running on those docker hosts.

One of the things that bug me in terms of cache as a service in AWS/Azure etc, it you're dedicating compute nodes to mostly use memory... the big win for memcached early on was utilizing unused memory on existing systems. You lose that when you don't have the caching services on the same nodes as compute/data/etc.

So glad I didn't read that article before I set up a Cassandra cluster on Docker to handle 1M requests/min in production. It might have discouraged me.

We are considering to use Realm to replicate our data out to all our docker instances. They would essentially work like databases local to each instance, and if the instance dies we simply spin up another one which will replicate out the same data again.

So far it is only at the experimentation stage for us, but it looks very promising. It is almost like having the ultimate cache (no network latency) right within each instance.

Hyper converged scale out object storage with docker engine on the same nodes could be your solution. Docker volumes support native NFS , rather stable IMO, there is no magic with docker volumes at all, your NAS, SAS, or distributed storage already implements all sorts of redundancy. And if you don't want to deal with anything just pay for DBaaS.

Why is running a DB in Docker so different from running in an OpenVZ VPS? There are millions of Wordpress websites and other PHP CMS-es hosted in OpenVZ VPS-es, running MySQL reliably.

Also, Discourse.org's default setup is PostgreSQL hosted in a Docker container, it also has probably 1-10 thousand live forums and is a reliable platform.

> I’ve seen DBMS containers running on the same host with service layer containers. But these service layers are not compatible according to hardware requirements.

> Putting your database inside the container, you’re going to waste your project’s budget. Why? Because you’re putting a lot of extra resources to the single instance. And it’s going out of control. In cloud case you have to launch the instance with 64GB memory when you need a 34. In practice some of this resources will stay unused.

For some software, resource consumption is of "fixed" size, plus temporary workload-dependent growth (e.g. application-layer processes, most of the time.) Whereas some other software will take up all the space available to it (like DBMSes.) The latter are what resource quotas are for.

Containers are not meant to be treated like "Unix binaries but more easy to deploy." Containers are just lightweight VMs that don't have to do screwy things with memory balloon drivers to efficiently pack many of those "fixed plus temp growth" workloads onto a host.

But like VMs, containers still need resource quotas to ensure they don't thrash one-another. You can avoid specifying quotas for your fixed-with-temp-growth workloads, to "oversubscribe" a host, and it'll work (similarly to oversubscribing memory-ballooed VMs.) But the "all the space available" workloads need quotas.

The author might be used to public clouds, where VMs have a "size" in vCPUs + memory and that "size" is charged for, and so might not think of picking an instance size for a VM as explicitly setting a quota. But when you set up your own hypervisor cluster, you still have to decide how big each VM should be, regardless of the fact that a bigger VM doesn't "cost" anything: a VM's "size" is the compromise you make between the needs of that workload, and the ability to "fit" other workloads alongside it on a host.

But, to go further: if you're designing "instances" and running dedicated workloads on them, you're very likely "doing containers wrong." (This is probably a provocative statement; stay with me.)

Containers are to container hosts as VMs are to hypervisors: in both cases, their architecture assumes that if you want resource-efficient deployment, you've got a big generic cluster of hosts, and your guests are loaded onto them using a bin-packing algorithm (taking into account which guests need what extra resources that are only available on certain hosts, etc.)

If you don't have a big generic cluster of hosts, then your only packing options will be necessarily sub-optimal. If your container hosts are real hardware, you're out of luck; if your container hosts are themselves VMs, running on some cloud provider, then costs will be heavily in favor of taking advantage of the cloud-provider's bin-packing by wrapping each of your containers in a separate VM and then deploying those VMs.

(Which is, coincidentally, what Amazon's Elastic Beanstalk does for you, and why it's not the same as Amazon ECS. ECS is for setting up your own "big generic cluster" of container hosts to bin-pack across; Elastic Beanstalk is for wrapping containers in VMs so that AWS will bin-pack at their abstraction level.)

isn't that what was said when VMs came out?

But I'd say it's not ripe yet to run a DB in a container for prod use. it's not an architecture issue but rather code maturity. DBs hold our beloved data and are more sensitive to hardware/system glitches which excersises much less frequent code paths in the DB.

We have been running databases in some form of containers for ages. Oracle DB on Solaris Zones, db2 in wpars. But yeah people have to re-implement these ideas poorly so now we have to deal with consequences.

So let your database write a journal. Data corruption problem solved.