Docker in Production: A History of Failure | Hacker News Reader

Docker in Production: A History of Failure

thehftguy.wordpress.com

575 pointsuser5994461287 comments

Author here.

A bit of context missing from the article. We are a small shop with a few hundreds servers. At core, we're running a financial system moving around multi-million dollars per day (or billions per year).

It's fair to say that we have higher expectations than average and we take production issues rather (too?) seriously.

I updated and fixed a few points mentioned in the comments. (Notably: CoreOS).

Overall, it's "normal" that you didn't experience all of these issues if you're not using docker at scale in production and/or if you didn't use it for long.

I'd like to point out that these are issues and workarounds happening over a period of [more than] a year, summarized all together in a 10 minutes read. It does amplify the dramatic and painful aspect.

Anyway, the issues from the past are already in the past. The most important section is the Roadmap. That's what you need to know to run Docker (or use auto scaling groups instead).

We have been running Docker in production for around six months now, with a moderate IO load. We are also using persistent storage, but our persistent storage is NFS-mounted inside the container, so 99.5% of the filesystem IO in the container does not touch AUFS. (We are stuck on an old kernel at the moment, but evaluating upgrading to either 3.x or 4.x.)

So far, this has been stable for us, and based on your post it sounds like it will continue to be a viable strategy as we move to kernel 3.x or 4.x.

You need to research Red Hat's offerings in this space (disclaimer .. 20 year RH vet). OpenShift is how the banks I work with are consuming kubernetes/container innovation. It's likely they have similar requirements, and they're loving it.

The banks I see (disclosure, I work for Pivotal) are consuming it through Cloud Foundry.

That said, there are quite a few mainframes that aren't going to be replaced any time soon, by either of OpenShift or Cloud Foundry.

You got me beat (Biltmore's a year short of 20 years, IIRC, oh yesteryear.) I tried searching for RH and banks and other than a landing page re: RBS and a few quotes it was particularly light on details. RH crushed it with the RHEL and JBoss model[1] (pretty much the paragon of "ethical open-source commercialization" IMO). Are these banks replacing pSeries AIX boxes at branches, WebSphere instances for the consumer to hit, or z13 mainframes with CICS or other more-than-five-nines-accessibility with guaranteed zero data loss? (Pardon my ignorance - not a snarky quip but rather genuinely curious as to what, if any, out of the box solutions RH is pushing in that sector.)

[1] http://www.forentrepreneurs.com/jboss-example/

To provide some experience I have as someone working for IBM implementing a solution which leverages docker at a large bank - no one I work with is naive enough to think that they can get away with containerising their current systems of record. Where we are significantly leveraging docker is on in-house bare metal clouds that we are using to build out middleware services.

We build all our images on top of rhel containers that have undergone hardening to meet internal + regulatory compliance. The end result is that to some extent we can say to developers "dont worry about getting your applications production ready right away, just write reasonably stable code quickly and we'll strip out all the stuff we dont trust you to do and handle it at the infrastructure level".

End result is that we can start to eliminate some of the unsuitable uses for traditional middleware systems like our datapower infrastructure that, while being great for specific use cases, is usually to difficult to work with for your average front end developer who doesnt give a damn about soap headers and broker clusters.

As far as our architecture leads are concerned, and I'm inclined to agree, docker is a great packaging format for developer outputs because it puts everything you need to run an application naked (ie no complex logging, HA, Networking etc.) Into a single versionable, reviewable, very much disposable asset. But it is not, and should not, be a replacement for proper systems of record that require any measure of stability.

Overlayfs like aufs is a layered filesystem but overlayfs got merged into the kernel. Aufs inpsite of a lot of effort was not merged. Before merge the module was overlayfs, after merge it was renamed to overlay.

I think its the responsibility of Docker users to understand the technologies they are using.

Docker has got into the bad habit of wrapping open source Linux technologies and promoting them in a way that makes it feel like Docker invented it. They did it to LXC and they are doing it to aufs and overlayfs. The HN community is far too vested in Docker to offer any real scrutiny and is very much a part of this hijack.

What is a Docker overlayfs driver? How is simply mounting a ready made overlayfs filesytem already present in the kernel or aufs a driver? These terms not only mislead but prevent recognition of the work of the authors of overlayfs and aufs.

They also require scrutiny as layered filesystems have tons of issues and the only way these can be resolved is engaging with the developers who most dockers users don't even know about. Docker can't solve these issues only work around them.

Addendum:

Did you ever wonder how much traffic can a site get from Hacker News?

Here's the number after 10 hours on the front page: http://imgur.com/1AJcYij

And reddit?

Can I ask (given the summary / roadmap at the end and your blog post about GCE's pricing): Are you moving to Google Cloud? If not, why not?

For what it's worth, I agree with you that a lot of the value of Container Engine comes from having a team ensure the bits all work together. That should be (one of) the positives of any managed service, but I think the Container Engine team has done a particularly good job in a fast moving and often bumpy space.

Disclosure: I work on Google Cloud ;).

GCP is awesome, I love the work you folk are doing.

Based on past discussions on HN and I feel google has huge terrible support for it's offerings. May be it will improve with time. But currently I am really concerned with running GCP in production.

I wonder if the problem is more that Google has got terrible support for it's free offerings - maybe for justifiable business reasons. However the damage to their brand when they want to sell business critical services is an unintended consequence. (Similarly I think killing Reader was justifiable from their perspective but the damage to their reputation amongst a vociferous and possibly influential demographic has probably outweighed the cost of keeping it going... They can't launch a service without someone popping up to remind us about Reader)

I miss Reader (though I'm amused that it let Feedly take off, and they're a Cloud customer!).

One thing often not considered ("Why not just have someone keep it running?") is that you really have to keep a team of people on it (in case there's a CVE or something) or at least familiar enough with the code to fix it. There's also the double standard of "you haven't added new features in forever!" (Maybe this wouldn't have applied to Reader though).

But, I agree if we could have kept it on life support somehow, we wouldn't have (as many) people asking "What if they shut down Cloud?!?". Conveniently, as people get a sense of how serious Google Cloud is about this business, even on HN I'm seeing this less.

I gave my opinion on this a few months ago: https://news.ycombinator.com/item?id=12323999#12324494

tl;dr it makes money, will make a lot more, they're going to support it.

We have true, paid support for Cloud Platform (https://cloud.google.com/support/).

As others (including downthread) have pointed out, Google definitely has a bad overall support reputation, but that's because nearly every (traditional) Google service is free and lacks this structured support model. Just like Cloud, if you're even a moderately large Ads customer, you get pretty good support!

Disclosure: I work on Google Cloud and want your business ;).

Given my experience with AWS. Even if Google has no support whatsoever, it's still better than the paid support from AWS ^^

We intend to execute the roadmap.

We have stuff that must stay on AWS, too dangerous to make a move now :(

We'll ignore the "switch to auto scaling" bit. Teammates have already started testing CoreOS & Kubernetes (on AWS) while I was writing the article. We'll figure it out and get it in production soon, hopefully.

We have a subsidiary which has no locking on AWS and could use an infra refreshing. They already have google accounts and use one or two products. They'll be moved to GCE in the coming months, hopefully.

Thanks for the reply; we'll see you as you get out of the lockin (feel free to ping me via email if you've got anything specific).

Why drop autoscaling though? Note that you can use Autoscaling Groups with Kubernetes on EC2: https://github.com/kubernetes/contrib/blob/master/cluster-au... and the team would happily take bug reports (either as GH Issues or just yelling at them on slack/IRC).

(Sadly http://kubernetes.io/docs/getting-started-guides/aws/ doesn't point to this...)

It seems that doing auto scaling or doing kubernetes may take similar amount of work (in our current state of affairs). We want kubernetes on the long term so we might as well go straight to it.

I suppose that auto scaling will be back on the table later, to auto scale kubernetes instances. Maybe.

Netflix uses Docker in production. While they don't use it for everything yet, they are transitioning to it. Netflix accounts for 37% of internet traffic or more in North America.

http://www.slideshare.net/aspyker/netflix-and-containers-tit...

Maybe you had trouble, but I think that those that aren't using containers need to be aware that many are using them successfully. Use of containers vs. VMs/dedicated servers can also result in reduced energy usage and can be less expensive.

I think people get hung up on what containers aren't and where they fail, rather than where they exceed.

But also be aware, that the traffic Netflix serves out of those containers is way less then 0.01% of the internet traffic in North America. The container cloud is their management engine.

The CDN that is actually pushing the traffic runs FreeBSD and would be using, if anything, jails.

Netflix's https://twitter.com/aspyker is traveling around talking about the benefits of containers.

On 28 Oct 2016 https://twitter.com/aspyker/status/792137233264418820 mentions "Lots of changes in the big cloud providers and container related leadership as of late."

Containers are becoming a big thing in the enterprise, which means they are here to stay.

Rackspace CDN-enables containers: https://support.rackspace.com/how-to/differences-between-rac...

Azure: https://azure.microsoft.com/en-us/documentation/articles/cdn...

etc.

Well, this person seems to have had a hard few months. I have a lot of problems with the post though:

- Docker do not own CoreOS and did not create the distro

- Docker did not write OverlayFS

- There are issues with storage drivers. Don't use AUFS for the reasons outlined. I've had no issues with Overlay2, but note there is also devicemapper (which Red Hat have poured a lot of time into) and BTRFS. And ZFS now IIRC.

- It's not correct to say Docker was designed "not" to do persistent data. You do need to be aware of what you're doing, and I suggest not running DBs in containers until you have the solved the FS issues and have experience with orchestration.

- Kubernetes has been around for a while now and is fairly stable in my view. Swarm isn't rubbish, but the new version is very young and you may run into issues in the short term.

- With microservices, when you are running lots of small containers, you have to expect them to crash now and again. It's just the law of averages. You need to design your application to accept this and deal with it when it happens.

If you hit this many problems with any given tech, I would suggest you should be looking for outside help from someone that has experience in the area.

I would like to add another factual error to the list - you can get linux 4.4 on debian stable with the backports instead of switching to an unstable release, and the current ubuntu LTS (16.04) runs 4.4 as well.

It seems that they started to use docker while it was changing a lot, I didn't have any issue in the past few months with docker, but I don't use it in production.

1) You've been running Docker for a few months. It could still be changing a lot, a few months is no proof one way or the other.

2) You are not running it in production. Poster is. All sorts of issues you don't care about on your dev box matter a lot in production.

IOW, your experience can best be summarized by "just trying out Docker in a totally different way and it works great!" Irrelevant to this post.

> note there is also devicemapper

You know that trick the post mentions about how you have to keep removing images to stop your machines running out of disk space? Yeah, removing images doesn't actually reclaim disk space on devicemapper (!!):

https://github.com/docker/docker/issues/3182

Oh, and devicemapper is the only natively supported storage backend on the AWS official AMI.

Yup, this is how I wound up with a 45 GB image on my development machine after futzing around with Docker for two hours. Does the same with OS X.

You have to use devicemapper in direct-lvm mode in order to get free space back. I agree it's not the best out-of-box experience, but so far we haven't had any problems running ECS on CentOS instances since switching to direct-lvm.

Yes, although loopback mode for devicemapper is the out of the box configuration for Docker even though Docker themselves doesn't recommend it for production. I had tons of issues w/devicemapper on CentOS 7 (kernel panics causing AWS to mark the host as unhealthy and kill it off) that I was finally only able to resolve by switching to AUFS under Ubuntu 16.04, which has worked pretty well for me. Knock on wood, AUFS has been pretty solid so far.

except when it comes to a many-level-image, AUFS performance degrades dramatically

well the article had some issue's especially around coreos, but it's good that somebody points it out.

Actually CoreOS produced pretty stable stuff: etcd, fleet. Actually with fleet/etcd combined with some cgroups i see barely a reason to use docker, I mean on Cloud's its actually another useless layer, at least for me, maybe not in Google scale.

container engines provide apis that orchestration softwares use to manage containers in all kinds of ways. In small setups with not much logistic going on i guess you can manage everything with fleet and some systemd units

Having followed the history of Docker since its LXC roots and it decided to write its own LibContainer library the only thing I want to point out is:

> It's not correct to say Docker was designed "not" to do persistent data.

Actually...looking at a lot of things "missing from" or "difficult in" Docker can more or less be traced back to Docker's overriding philosophy of container runtimes. Things like not having an init process and optimizing user experience around single "Run" commands.

Having relatively recently done a container retrospective, following the path of:

OpenVZ -> LXC -> Docker -> now (CoreOS, Rancher, LXD, Mesos, etc.)

I can't help but feel as if some of the "limitations" of Docker were more or less designed.

I'm not saying it's a bad thing, being a Python enthusiast I see value in simplicity/having only one way of doing things...but I just wanted to point out that where you draw the line for what is considered "designed" starts to get blurry in areas of Docker's chosen implementation.

Don't forget that DotCloud/Docker was originally basically a clone of Heroku, where "12 factor" dictated that all persistent data was stored in (non-containerized) Postgres.

That's incorrect. Dotcloud ran both stateless and stateful services on the same container fabric, unlike Heroku and every other PaaS. Specifically you could run a managed containerized postgres, mysql, mongodb, redis, rabbitmq, and custom stateful tcp services. Docker was a result of that design.

Docker was specifically designed to NOT impose a model of ephemeral containers. For example, by default 'docker run' preserves all data.

We use https://github.com/l3nz/whaleware for persistent, database-oriented workload with persistent data. We used to keep persistent data on a data volume, but now mount a native directory instead (2x faster).

Might not be the True And Approved Docker Way, but who cares? it works well, and has been in production for a number of years now. No data loss ever.

> If you hit this many problems with any given tech, I would suggest you should be looking for outside help from someone that has experience in the area.

I suggest using something that works for you and your business instead. If the only thing that works is $SOLUTION, but you have these difficulties, then yes by any means look for help.

Edit: Ok, I feel compelled to extend my reply as I've been downvoted without comment, which is terrible.

Here's the thing: You're not forced to use Docker just because it's the next hyped thing. It is great and yes, I use it. However there is more than way of doing things and the only reason to choose one technology over another is if it's better for the business.

Anything else is just ignoring reality.

I did not downvote you but it sounds like the author works at one of those horrible places where developers get to do things that make no sense in production without undergoing an operational review. He probably cannot change that (change would have to come from devs in a company like this), so your advice boils down to finding a new job.

More specifically, it sounds like it's one of those places where developers got to do things in production that didn't make sense.

Anonymous downvoting is pretty popular around here. I agree it's a very rude thing to do. Also, just an FYI, I was scolded the other day for acknowledging the downvoting in an edit.

Edit: case and point

Please don't be surprised by this behavior. It's in the guidelines to do this and is what most of us would prefer.

> Please resist commenting about being downvoted. It never does any good, and it makes boring reading.

> Please don't bait other users by inviting them to downvote you or announce that you expect to get downvoted.

By "most of us" who exactly are you referring to? I wasn't aware there was some kind of user survey etc that was used to determine that stuff. I thought one person just added that to the bottom of the comments guidelines. Was there a vote or something? I'm very curious how you know that is what "most of us would prefer".

Can you point to any evidence that people enjoy talking about downvoting? More people are allowed to upvote than are allowed to downvote so if your post is negative, you're clearly doing something the majority of people reading it disagree with.

Can you point to any evidence that people enjoy being down voted without an explanation? Clearly the fact that there has to be two entries in a fairly short comment guidelines points to the reality that many people have expressed that they do not like being down voted without any explanation.

And since it is such a frequent occurrence, would that not point to the fact that people do enjoy talking about being down voted?

"Issues with storage" is an understatement. Linux has major issues with storage. I see no way forward for Linux in that area unless they adopt ZFS.

How many filesystems has the Linux world created in the last 20 years? Of those how many are rotting piles of tire fire?

Filesystems are hard to get right.

ZFS is the way forward. For cross compatability, for reliability, for stability, for lots of use cases.

Btrfs is another tire to throw on the burning pile of other filesystems that won't work out in Linux.

Let the down votes begin.

But Linux has some major architectural issues that eventually the Linux faithful will have to admit to.

Containers.

Filesystems.

Observability. Though bless him Brendan Gregg is trying his hardest to help here.

Portability.

And this is what is so painful for the rest of us to watch or deal with. This inability for the Linux folks to admit these glaring architectural problems. Problems they refuse to look outside their bubble to see how others solved these problems and adopt solutions that have already solved these problems. They just want to continue to bury their heads in NIH soil. And double down on trying to solve these issues, poorly, on their own without any awareness of how others have solved these issues. People outside Linux land just might have the right ideas on how to solve these problems!

But I expect no one in that camp to acknowledge this and down vote away.

Sigh.

Sir, you have spoken well. What OS/distro should I be looking at?

CoreOS, as stated by the roadmap of the article. /s

Enterprise: Solaris and use zones. Internal/Personal/POC: FreeBSD 11 and use jails.

Both get you ZFS and something lighter-weight than VMs. Need Linux? Bite the bullet and use VMs.

I agree with this. SmartOS would be my choice over Solaris. I think FreeBSD is getting there. With things like Tredly. The building blocks are solid and available they just lack some polish.

So if you have the skill FreeBSD. Otherwise SmartOS makes a killer setup for containers unmatched by anyone else.

A warning, though: don't try to use FreeBSD with SCSI tape drives, in my truncated experience because of what I found experience a few months ago, the driver is capable of writing without error data it then can't read back (but can be read back by Linux).

Maybe not so important in the container context, but sooner or later, somewhere you need persistence, and tape offers certain persistence features you pretty much can't get elsewhere, especially at its media price points.

(Plus, if they got this wrong, when it's really not that hard to get right (I've done SCSI at this level before), I wonder what else they have.)

well, checkout hypercontainer (https://hypercontainer.io/) that runs on Linux, uses VMs and performs like containers.

No killing yourself necessary, really ;)

Solaris is discontinued but Illumos and SmartOS have taken up the torch.

That would come as a surprise. Solaris is alive and well thank god https://www.oracle.com/solaris/solaris11/index.html

See? Solaris is dead

And this is what is so painful for the rest of us to watch or deal with. This inability for the Linux folks to admit these glaring architectural problems.

Alright, the suspense is killing me: what OS is the one that gets it right? I ask this to "the rest of us" as you put it.

From the username, x86 BSD perhaps?

SmartOS, FreeBSD, Tredly, etc

Ah, you must be from the alternate universe where most of us have heard of more than one of those!

What you've heard of, and what's workable and useful, do not necessarily intersect.

Tredly is "just" a FreeBSD distribution. And Linux can't have ZFS in-tree because CDDL.

It is not just a version of FreeBSD. The base is FreeBSD but the tooling for containers is not stock. It's value added bits that are shaping up to be very nice for containers on a more stable, secure, and storage friendly stack.

Yes, but the storage parts which the discussions are about are from FreeBSD

NixOS intrigues me as an attempt to create a completely deterministic build.

I can see a lot of inflammatory content here - vague accusations about "tire fire" filesystems and how Linux users are "faithful" zealots, blind to the truth, that will have to "admit" their folly and implicitly humble themselves before "the rest of us", whoever they are. And baiting a bad response, as you ask people to "downvote away" as if you want to inspire anger. All of this is needlessly confrontational. You don't need to insult your opposition to make your point.

You may view them as inflammatory. But it's coming from a POV of pain having to deal with broken architecture, piss poor portability and general pain of hacking around poorly engineered Linux architecture.

Frustration, pain are more appropriate terms from my pov than inflammatory. But ymmv.

It's like "Come on folks! You can do better than this! Please! Just stop! Think! Please. I'm begging you."

Regardless of whether your point is true or not (I agree that the state of Linux is suboptimal), this kind of attitude in which you spend more on venting your emotions than on actually describing issues, will not help you get your point across, and will not help making the world better. In the worst case you may even give BSD people a bad name by making the public perceive them as a bunch of emotional haters.

> ZFS is the way forward. For cross compatability, for reliability, for stability, for lots of use cases.

I'm not entirely convinced we should settle on ZFS just yet. It's fantastic and quite possibly the best option right now but it has a few limitations:

- The Linux implentation seems to have issues with releasing memory from the ARC back to the system

- The only way to expand a zpool is to add a new vdev. Pools are essentially a RAID0 of vdevs so if a single vdev fails, your entire server fails. You can mirror or RAID within a vdev but this means that the reliability of a vdev is the reliability of your entire pool. The problem here is that you can't just add 1 or 2 new disks since adding a 1 or 2 disk vdev would be data-suicide. For smaller servers, this is silly.

BTRFS looked like it was getting there but it proved buggy and unreliable. Personally, I'm waiting for bcachefs: https://www.patreon.com/bcachefs

There's nothing wrong with a 2-disk vdev. Mirrored instead of RAIDZ is probably the best choice for "smaller servers."

I use stripped mirrors for most of my ZFS deployments. Disk is cheap. If you want more resilience do a three way mirror.

I strongly disagree. If I have an 8 disk server at home with mirroring, the pool could only take a single disk failure. The usable space would be 50% of the total disk. Further, I'd have to buy any extra disks in pairs.

If instead I ran RAID6, I'd have 80% of the disk available and I could add disks in single disk increments.

I think ZFS makes great sense for businesses that can throw money at disks but for smaller businesses or home servers it's kinda bad.

You also wouldn't get the same I/O performance a stripped mirror will give you. You can get extra resiliency using a three disk mirror.

Disk is cheap. There's no reason to design like it's not.

Disk is at least ~$0.023/GB. Whether it's "cheap" or not depends on how much of it you need.

Plus, disk may be cheap but servers to house it are not (the kind of servers you'd run in your house, I know you can get cheap SC846 off eBay).

Servers can be inexpensive. It really depends on your use case. You don't have to drop 50K on a box to make a killer ZFS storage array.

My scale starts at 40PB with a "P". Disks aren't cheap.

I run a FreeNAS device at home (ZFS underneath).

It was kind of a pain to configure (albeit quite flexible) but it's been pretty nice overall, already survived 1 disk failure and a capacity upgrade (during which I had to resilver after every individual disk upgrade, which was time-consuming, but after the last disk got upgraded, the extra space finally showed up)

I think it's not a very good idea to design storage solutions around how inexpensive it is to add capacity to them. Your 9+-drive RAID6 is going to take forever to rebuild; 4+ mirrored vdevs (or mirrored RAID of course) will not be a problem at all.

> Personally, I'm waiting for bcachefs

tires++

;-)

I'd extend your point on zpool expansion to ZFS's general inflexibility when it comes to reshaping pools or rebalancing ZRAIDs.

I'm sure it's not an issue in enterprise environments, but I've personally been bit by ZFS's inability to shrink or rebalance pools (even offline) several times now through personal use.

Those sort of use cases need to be handled if ZFS is to become a more general use filesystem.

Many Linux users in particular are pretty well aware of the shortcomings of Linux's filesystem scenario, as well as the fact we never had features like DTrace. (I still think the over-proliferation of Linux tracing tools that fit 10,000 different case is a bit ridiculous.) As someone who worked on a distro, I've been acutely aware of many of these things, for example, as have our users (because they inevitably come to us for advice, suggestions, or bug reports).

I'm not sure what reality you live in where Linux users like, actively deny things like lacking an appropriate COW filesystem with low latency, or things like the fact it took us a decade to get only close to DTrace. Your own mind? 6 month Ubuntu users? Internet forums where you talk to other BSD users and nobody else?

> Let the down votes begin.

Don't worry, I'm sure people will oblige.

Ah yes, ZFS the saviour.

Just as long as it does't get anywhere near capacity.

I assume you are referring to the same limitations almost every FS on earth to date suffers from when you consume about 90% capacity?

What a useless comment. If you are running out of capacity adding disks to a zpool is incredibly mind numbingly easy.

"zfs add poolname mirror disk1 disk2"

Every filesystem has performance hit the floor when you consume all available capacity. This is not unique to ZFS but to well, reality.

No I'm talking 100% capacity, shit happens and I do t want to fight the fs when it does.

100% is a lot better than ~80% (looking at you btrfs)

Init systems.

I like ZFS,

However XFS is perfectly good for storage. I've used it on 15pbs worth of fileservers with no particular nonsense.

I wouldn't touch BTRFS with a barge pole, not because of any inherent stability issues (although that is a big factor) its the utter nastyness of the tooling.

ZFS is a joy to setup, and its simple, logical and the man pages tell you useful things.

BTRFS, not so much

My favorite XFS workflow:

1. filesystem is corrupted, once again

2. try to repair it

3. oh right, the repair tool can not replay the journal

4. try to mount it

5. admit after 3 hours of nothing that the journal-replay code triggered on mount actually can really not deal with corruption

6. reboot the server to get the filesystem unstuck

7. rerun repair, this time throwing away the journal

8. look at the empty filesystem with everything in lost+found

9. restore from backup

The team I'm in only runs 1'400 servers, yet this happens regularly.

point taken. I have had xfs_check run out of memory because $reasons. This has reminded me of that. But then we also had lustre, so you know, by comparison XFS is a paragon of stability (We also had clustered XFS....)

But, mostly it happens because the fileservers dont have UPS's (The pipeline tools are almost exclusively COW, and backups are tested many times a week.)

I think we paid for support on XFS from either redhat or SGI, but I can't remember, I left that place a year or so ago.

I've never had the balls to run ZFS on large(100tb+) arrays. Last time I tried the way slab handling was translated from solaris cause many problems (but that was more than 4 years ago. ) Plus the support is a bit odd, you either go with oracle(fuck that), or one of the openZFS lot.

To get the best performance/stability you ideally need to let ZFS do everything, instead of letting the enclosure do the raid 6 (4 * 14 disk raid 6 with 4 spares). This of course is a break from the norm and to be treated with utmost suspicion.

Have you seen the GPFS raid replacement? thats quite sexy.

I run ZFS at home, because it works, and has bitrot checking.

Yeah, I flagged the article for being grossly inaccurate. I've had a lot of frustrations with Docker myself, and will happily switch to Rkt after it matures a bit more. Docker surely deserves a lot of criticism, but it should at least be valid and well-informed.

I think Docker can somewhat be seen as a generalized term like Linux. Linux doesn't write EXT4 but it is often referred to as "Linux filesystem".

Docker has been actively trying to become to household brand name for containers so it's difficult for me to give them a pass with your line of reasoning.

> Re Persistent Data: You are probably going to have a tough time if you try to do persistent data. Containers really excel at stateless tasks(or temporary state like caching). Certainly it's doable but for someone early in their adoption the proper toe in the water is probably the stateless business logic not the state layer.

That is Dockers problem, they even want to be on Windows, I mean come on. But thats docker the company for you.

Can you blame them for wanting to break more into the Microsoft ecosystem? Besides, Docker's company mentality aligns pretty closely with a company like Microsoft, there's probably a lot for them to learn. A huge plus from a business perspective, but it's easy to see why it wouldn't sit well with open source advocates.

Yup I blame them for that. (I is freedom software advocate)

Not a company of good taste in my book.

I thought the whole Windows thing was totally stupid (I is also freedom software advocate). I've been a big fan of Docker for about three years, put loads of stuff into production at a handful of different companies, most recently using native Swarm features that came with 1.12. Personally, I thought the Windows thing was a gimmick. But what I see (as someone that has access to the Docker community through open source) is this enormous group of people who are chained to Windows technologies. And for years and years they've watched all the cool kids play with the cool toys, from the sideline. And what Docker-for-Windows has done -- with help from a new kind of Microsoft -- is expose all of these very eager people to containerization and microservices and open source and free software more generally. Having been exposed to loads of these people (and there are an enormous number of them), I think it means a lot of important things about bringing open source and free software into the fold for these people. Docker-for-Windows opens the door for a yuge number of developers to appease their enterprise leadership by sticking with the Microsoft brand, while also having the whole world of free software at their fingertips. I can see this world on the horizon in which open source and free software concepts have broken into the Microsoft ecosystem, and I believe it comes waving the Docker flag. I am still extremely suspicious of Microsoft as a company, and wouldn't pay for Windows if my life depended on it, but you can't say they aren't doing some interesting things over there at Microsoft lately.

There is time and resources spent on improving free and open source software, and then there is time and resources spent improving proprietary software for another business.

Pick one.

Now, that's not the reality even slightly.

Minor: Ubuntu 14.04 can run an LTS 4.4 kernel via the linux-generic-lts-xenial package.

> It's not correct to say Docker was designed "not" to do persistent data.

Any monkey can run a bunch of stateless systems. It's the stateful ones that have the worthwhile problems.

> - Docker do not own CoreOS and did not create the distro

Not just that, CoreOS is basically made to push systemd-nspawn as a container framework as hard as possible.

rkt which uses systemd-nspawn was started after docker shifted focus from just the containers to the whole environment

Is that a problem?

This is why DevOps became a thing. It's complicated managing systems at scale. If you were using these tools and aren't at scale, well, that sucks.

Unplanned downtime is the main drawback to both hosting your own OS's and using leading edge tooling for your critical systems. It doesn't matter what the underlying system, stuff like this happens. This stuff is complicated. You will find major warts in every new system. Everything will break, often, leaving your users stranded. It takes a very long time for software to mature. [0]

That's why you see engineers with 15+ years of experience patiently waiting for new technologies to mature before putting them in place in our critical infrastructure. Sure, we'll play with them, but putting new tech in production that could easily make your entire system unavailable is too risky. Yes, three year old tech is new.

[0] http://www.joelonsoftware.com/articles/fog0000000017.html

Couldn't agree more. This is the reason I haven't started to recommend Docker for production use yet to our clients. We are pretty happy with only having a single JAR deployments for all of the services, running with separate users that cannot access anything else than the service folder. I am considering using runC probably starting early next year, seeing the amount of problems with dockerd makes me nervous about introducing it to any team.

It takes a fair bit of experience before you can recognise the warning flags, though.

Most folks don't realise how simple their requirements are. Most folks need to put files on a server and start a process. What they don't need is a whole mass of rickety scaffolding and control plane infrastructure that gets them halfway (and only halfway) to running an in-house PaaS.

So having kicked the tyres on Docker and been appalled by the messy design and dreadful tool quality, I went back to using OS-native packages and isolating our microservices via (gasp) user IDs.

Judging by their product direction I suspect Docker's board want them to challenge VMware, which (rubbing my crystal ball) suggests to me that their future is a bloated and overcomplicated piece of Enterprise, targeted at companies who think you can buy devops.

Well container came out of slimming down VMs.

Meaning that rather than moving a whole OS from hardware to hardware, or spin them up or down as load required, you would spin up or down processes and their runtime requirements from a known functioning image.

Thus removing the overhead of the VMs hardware emulation and the need to run so many kernels.

The response from the VM world to containerization seems to be to push unikernels. Effectively going back to the DOS days where you have just the minimal kernel and userland, all living "happily" in the same (virtual) memory space.

Well, thats what cgroups and chroot is for, but then docker is just a fancy wrapper.....

As for performance, if its critical then get a real server, if you've got decent change management (ie ansible, puppet et al with no manual intervention) Then its no effort.

(seriously network boot, and vlans are your friend here.)

> This is why DevOps became a thing.

Huh?

> This is why DevOps became a thing.

How do you define "DevOps"? Because configuration automation (e.g. cfengine) and deployment automation (e.g. Capistrano) were more or less widely known and (less widely) used before the word (portmanteau) was popularized.

Docker's reliance on overlay filesystems is one of the biggest problems I have with Docker. Stacking opaque disk images on top of each other just isn't a great design, and it makes for a cache strategy that is all-too-often invalidated (because a Dockerfile is linear, there is no dependency graph). The file system is just the wrong layer of abstraction for solving such an issue.

If you change paradigms from the imperative sequence of mutations in a Dockerfile to a declarative specification that produces immutable results and a package dependency graph structure, you get a much better cache, no need for disk image layering, a real GC, etc. For example, the GNU Guix project (purely functional package manager and GNU/Linux distro) has a container implementation in the works named call-with-container. Rather than using overlay file systems, it can just bind mount packages, files, etc. inside the container file system as read-only from the host. Not only is this a much simpler design, but it allows for trivial deduplication of dependencies. Multiple containers running software with overlapping dependency graphs will find that the software is on disk exactly once. Since the results of builds are "pure" and immutable, the software running inside the container cannot damage those shared components. It's nice how some problems can simply disappear when an alternative programming paradigm is used.

https://www.gnu.org/software/guix/news/container-provisionin...

One can get very similar experience with Docker. The idea is to make single Docker image with all the software one needs on the server and start all containers using it with read-only flag. Then one use volumes for persistent data.

That's not really the same thing. What about different applications that share dependencies? With Docker, they would have to share an exact subset of a linear Dockerfile to take advantage of any deduplication, so you're back to image layers. It's telling to me that when proposed solutions involve completely avoiding one of Docker's primary features that maybe Docker isn't very well designed to begin with.

With Guix, any sub-graph that is shared is naturally deduplicated, because we have a complete and precise dependency graph of the software, all the way down to libc. I find myself playing lots of games with Docker to take the most advantage of its brittle cache in order to reduce build times and share as much as possible. Furthermore, Docker's cache has no knowledge of temporal changes and therefore the cache becomes stale. Guix builds aren't subject to change with time because builds are isolated from the network. Docker needs the network, otherwise nothing would work because it's just a layer on top of an imperative distro's package manager. Docker will happily cache the image resulting from 'RUN apt-get upgrade' forever, but what happens when a security update to a package is released? You won't know about it unless you purge the cache and rebuild. Docker is completely disconnected from the real dependencies of an application, and is therefore fundamentally broken.

Docker needs network only when one uses Dockerfile for deployment, a rather bad idea. Instead Docker images should be used. It allows to verify them on development/testing machine before deployment. And with this setup all "bad pieces" of Docker are located at developer's notebook. In production everything is read-only and shared among all containers.

As an LXC user since very early days, I wrote something similar to but architecturally far more generic than docker, begun earlier (~201-2015). It's called cims. It was explicitly designed to be portable to *BSD as well as arbitrary storage drivers (docker added this later), cloud provider interfaces (VM only, AWS/GCE style full APIs, existing server with LXC or docker in place, etc.), and between arbitrary logical machine and service topologies. See docs @ http://stani.sh/walter/cims/ and early architectural forethought @ http://stani.sh/walter/pfcts/ Scope was arguably broader (ease of system administration, explicit need for existing CI/CD process integration), and to date I still believe the architectural paradigm is superior. Unfortunately my former employer decided to ask me to pay them to release the code, so I can't release it open source. But the high level docs have been open since ~early so knock yourself out on a clone. I may even write one myself in future.

Storage-wise, it had LVM2, ZFS, and loopback drivers. Never bothered with overlay, just used a generic clone API to the storage driver to spin up identical VMs before modification. Very easy with snapshot/thin-provisioning capable backends, like LVM2 and ZFS. Loopback just used cp, but because you could have these in memory they could also be very fast (but memory-hungry). Cloud-wise, we'd done a few but I was also iterating an orchestration system based upon internal requirements (high security/availability) and existing, proven solutions (pacemaker/corosync). It was designed for fully repeatable builds, something docker only began to add at a later date.

That's odd, we've been using Docker for about a year in development and half a year in production (on Google Container engine / Kubernetes) and haven't experienced any of the panics, crashes yet (at least not any we could not attribute as a failure on our end).

Clearly, Docker itself has a a few annoyances, like no garbage collection and poor tagging process, but we're using it on a daily basis, and nobody complains about it (nothing a few bash scripts can't fix).

Finally, the biggest benefits of not just docker but containers in general are consistency and immutability which brings me to the final point I agree with the author: Your database shouldn't be containerized. It's not your pet (from pets vs cattle), it's your BFF.

OP wasn't using orchestration.

He wasn't using orchestration, but said Docker Swarm is terrible. How did he come to that conclusion.

For the record, orchestration in Docker (1.12.x) is actually quite good. Its built-in service discovery and mesh networking make setup a breeze.

Yeah, for every article complaining about how Docker in production is just so terrible there's a huge group of people who have been running it for a while and have no problems at all.

Nobody writes rant posts when things work, I guess.

Here's one, https://eng.uber.com/dockerizing-mysql/

I definitely try to highlight the success stories, but there's a critically different way that audiences react to "here's why everything went great for me" on Hacker News, though I think that is understandable. Too many times they're customer success stories, or case studies from the marketing department, and many don't care for those.

What is wrong with DB in container? As long as one uses host volumes for persistent data, docker allows to isolate DB setup rather nicely.

Yeah I'm wondering this too...

Agreed and confused. Interested to find what would be wrong with this? (from my docker-compose.yml under the db service) volumes: - ./data/postgresql:/var/lib/postgresql/data

Read my reply below

Reverse question that no-one seems to ask: What's the advantage of running a DB on a container?

Spinning a new db in continuous integration for integration tests.

How do you host your DB on GKE? Are you saying there's something wrong w/ a DB pod that writes to a PV?

We host PostgreSQL on GCE with two separate instances (master/standby) replicated with repmgr (http://www.repmgr.org/). Then we have a service defined with custom endpoints that point to these two instances.

There's plenty wrong with a Pod that writes to PV - if your Pod/Container somehow gets corrupted, you're left with a corrupted database. It's unlikely, but as the origintal article states, Docker is young and things happen. Are you willing to risk it?

So in every case you need to ocnfigure replication and master selection. Are you willing to trust the pods to do the right thing without manual intervention on data that is the core of your business? I'm not.

What exactly do you mean by "container get corrupted"? One should always use the read-flag to run Docker containers and use explicit volumes for persistent data. With this setup a container can only get corrupted if a read-only bind mount gets corrupted, and this will be a bug in kernel, not Docker.

1) The conclusion of the article is to use Google Container Engine because Google are the only one who might have managed to get a working ecosystem around Docker and may be able to maintain it.

2) As disclosed by other comments, Google Container Engine doesn't use Docker. It uses containerization technology made by google for google, with a docker-interface added on top.

That's why you don't have problems with Docker. You're not using Docker at all! ;)

I work for a large enterprise 100K+ employees using Docker in production [1]

It's certainly had its challenges (different versions of Docker clients causing havoc with different manifest versions, race conditions in the engine etc etc), but this article takes things a little far.

If you take pretty much any new technology like this make it a base part of your platform, you'll want support from a big vendor. To think otherwise would be naive.

There is definitely a debate to be had about Docker Inc's approach to production/enterprise rollouts though, but as a technology I'd say it's developing pretty much as I'd expect.

I've also seen Docker succeed in production despite the technical challenges. If you don't like the heat...

(1) See my blog: https://medium.com/@zwischenzugs, especially: https://medium.com/zwischenzugs/a-checklist-for-docker-in-th... and https://medium.com/@zwischenzugs/docker-in-the-enterprise-ec...

The most important challenge that Docker faced was where it wanted to position itself, IWO, as a solid stable building block for container infrastructure, or as a all-in-one-alike container solution. Now that Docker has made its decision to push Swarm mode, it is getting better for productions. There are instabilities here and there but it will come around sooner or later.

From the article: "The impossible challenge with Docker is to come with a working combination of kernel + distribution + docker version + filesystem. Right now. We don’t know of ANY combination that is stable (Maybe there isn’t any?)."

What combination of Docker version + distro + kernel + filesystem are you running, and how stable is it?

RHEL7.x - docker versions 1.8.x-1.10.x, mostly ext4. Stability is OK, aside from the things I mentioned.

One thing that bit me is that on RH 7.1 you _should_ use ext4[1], but on RH 7.2 you _need_ to use xfs[2]

[1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterp... [2] https://access.redhat.com/documentation/en-US/Red_Hat_Enterp...

"The only way to clean space is to run this hack, preferably in cron every day"

https://github.com/spotify/docker-gc

"Many attempts can be found on the internet, none of which works well. There is no API to list images with dates"

http://portainer.io/ seems to be able to do it, docker images lists it... I mean, I don't want to call bullshit without knowing the full story, but...

"So, the docker guys wrote a new filesystem, called overlay"

https://github.com/torvalds/linux/commit/e9be9d5e76e34872f0c... written in 2014 by the linux kernal team / Linus, but OK cool story

"It affects ALL systems on the planet configured with the docker repository"

OK, this is why people use AWS' container repo, or they use the open source software and maintain their own repo... this happens with any of the public repo services, and it was 7 hours.

"The registry just grows forever"

S3 is a supported backend, highly would advise anyone running their own repo to use it (or have a similar expanding-space story). There's also a config flag for the repo to allow removing images. Obviously wouldn't want to use this if you're hosting a public repo but internally, go for it, it's off by default, seems sane enough.

"Erlang applications and containers don’t go along"

I'm certain that people are running erlang containers successfully.

"Docker is a dangerous liability that could put millions at risk. It is banned from all core systems."

Daily FUD allowance exhausted.

I guess the tone of this article really bugged me because there's obviously a point to be made from this experience that running Docker is more difficult & error prone than it should be. And maybe it wasn't a great fit for this company culture. But this article crossed the "we didn't have a good experience, therefore, it's BS software" line, and frankly, that attitude may very well have been to blame for the lack of success just as much as docker's shortcomings were...

> This script is intended to be run as a cron job, but you can also run it as a Docker container

I mean, he's right on the cron part. That's how all the various open source docker cleaners are working.

Sure, it all hinges on cleaning up on a schedule, however you want to do that, cron is probably pretty popular. But the "hackiness" was due to a rudimentary approach, not because it inevitably needed to be a hacky process.

Right, the whole argument doesn't make sense. What should Docker do instead? Removing containers after they stopped so you have no idea what happened when it crashed? "Reuse" the container? Apparently somebody didn't grasp the concepts* and therefor considers all this BS.

*) Docker could maybe do better here, but tbh - I think documentation and getting the word out about how things work is one of the things Docker Inc does pretty ok.

Take a look at Rancher also. It is a bit more mature project, and it did save me some headache. To be honest I didn't use Docker everywhere and everywhen but where I did, do real issues.

Also there is DCGC from yelp!

https://github.com/Yelp/docker-custodian looks awesome, thanks for the tip! Rancher looks great too -- would love to know if you looked at other options (kubernetes etc.) & how Rancher won out

I'm writing sick from bed so my posting capabilities are slightly diminished :) .

Rancher itself is not an alternative to Kubernetes. It can control a Kubernetes, AWS, Mesos or very simple Cattle cluster. We started with Cattle with a plan to one day use something like Kubernetes, a step at a time, because we kinda felt intimidated (I left that company) by articles like these.

FYI, Rancher can use Kubernetes for orchestration (Mesos or Swarm too)[0]. No idea how well that works though, I am using it with their own Cattle and that is more than enough for my lowly needs.

I looked at Kubernetes before Rancher but it was hard for me to really understand where to start and how it all fit together. Now that I understand much better what I am doing I should probably take another look at it.

[0]: http://rancher.com/rancher-now-supports-kubernetes/

I love RancherOS - why do you think it's so undervalued compared to CoreOS?

> Docker Issue: Can’t clean old images

The worst part of this problem is when you do run out of disk space docker's image/container database often gets corrupted resulting in the great cryptic 'could not find container entity id' error. The only fix I'm aware of is rebuilding all your images or mucking around with the docker sql database. The lack of basic features to manage the problem creates a worse and harder to troubleshoot problem.

It's an easy enough problem to avoid in the first place so it's really not big of a deal. It hurts docker as a company more than the users in the long run because it makes them look bad. There are a lot of other small things like this with the docker cli tools especially. They aren't deal breakers but they are definitely eyebrow raisers. For example if you mess up your docker run command it usually just fails in strange ways instead of telling you what's wrong with your command.

I'm actually pretty happy with docker overall but they really need to fix some of these basic usability problems.

I also work in HFT, and I also attempted to use Docker in production and encountered all sorts of problems (different from what are described in this article, though).

However, as it turned out, these problems had nothing to do with Docker. Containerization is a great idea, and Docker's approach is sane (one container per service — fine).

The offender was docker-swarm. Like many others, I chose it as the default container management approach (it is written by the same team, so it should work the best, right? Wrong.) Docker-swarm is indeed buggy and not ready for production, or at least it wasn't 10 months ago. And if you use "one container per service" approach, orchestrating large groups of containers is a necessity.

Then I discovered Kubernetes, and became significantly happier. It's not without its own set of quirks, but it orders of magnitude smoother experience compared to docker-swarm. And it works with AWS just fine. (I didn't get to use DCOS, but I heard some nice things about it, too).

Tl;dr the source of all "Docker is not ready for production" rants seems to be docker-swarm, at least from my experience. Use Kubernetes or DCOS and everything will be so much better.

Here is the smoking gun, showing that this organization shot itself in the foot:

> Docker first came in through a web application. At the time, it was an easy way for the developers to package and deploy it. They tried it and adopted it quickly.

Unless you've got a lot of spare time on your hands, it's never a good idea to adopt unfamiliar tools where they're not needed. Stick to your traditional deployment techniques unless you're equipped, and have the time, to take a roller coaster ride.

Docker is young software.

That said, it seems the author did experience many legitimate breaking changes and abandoned projects. It would be great to hear a perspective from someone on the Docker team about how breaking changes are managed, how regressions are tracked, how OS packaging is handled (e.g. the dependencies the author mentioned wrt filesystems, as well as the bad GPG signature), etc.

> Unless you've got a lot of spare time on your hands, it's never a good idea to adopt unfamiliar tools where they're not needed.

That's so broad, it's meaningless. Nothing is ever "needed". Given enough time, you could run everything on a stack limited to technology released before 1995. And without "adopting unfamiliar tools", they'll remain unfamiliar forever.

True, however if you mix in maturity I think you can come up with a reasonable formula for new tech adoption. I only just started using Docker this year because I was giving it time to mature. When the Docker for Mac Beta came out I started playing with it because it promises to solve real problems with using Vagrant. If it was more stable I might jump in and start replacing Vagrant boxes faster, but because of the memory leaks / freezes / CPU overhead, I'm not pushing forward aggressively. Meanwhile, let the cool kids bang their heads against the wall until it stabilizes a bit more and then I can reap the rewards with a lot less pain.

How is it meaningless? The purpose of a production system isn't to run experiments. It is to stick to what is currently known to be reliable and stable. If that means sticking to old technology then so be it.

To me the article shows more about issues with the OPs organization and approach than Docker.

And that while I think Docker does a really crappy job and I went through so much pain myself. But the whole article shows lack of understanding and approach. I wish we should discuss more constructive about all the issues Docker indeed has..

Like the author I've also been running Docker in production since early 2015. Given the 'HFTGuy' part of his email address I'll guess I don't do anything nearly as edgy as he has to do on a daily basis, so it would be tiresome to just say that I haven't had the problems he's experienced. A lot of his issues probably do stem from his organization pushing the limit and wanting to run very specific configurations for performance reasons.

The one thing I do want to respond to is the notion that Docker cannot run stateful apps. Among the stateful apps I have run on Docker in production: elasticsearch, redis, postgresql, mysql, and mongodb. Containers and application state are orthogonal topics. The contained process is reading and writing a file system on persistent media just like all the other processes on the host and that file system doesn't mysteriously disappear when the contained process exits. Naturally stateless apps are simpler. They're simpler to run on VMs or bare metal too. When you need data in a container you do the same thing you do when you need it on a VM or on bare metal: you mount appropriate storage for it.

The main issue with most stateful applications isn't the existence of the state. The main issue is that if you have state then its probably important, you probably want to make it durable and highly available, you probably want clustering and replication, and now you're into discovery and peering, and that can be an actual issue with Docker, especially for finicky beasts like redis.

I haven't seen anybody speak to one of his points and I'm curious about what HN thinks: if you're already using a cloud provider that provision's instances of the exact size you need, what benefit do containers bring?

I'm excited by containers and all, but this point has always stopped me from going forward. If I were a huge company running bare metal, then yes I'd want to squeeze every last ounce out of my hardware. But if I'm running inside a hypervisor and somebody else is paying the cost, what are the benefits?

As for the article: it comes across as a blend of frustration and real technical challenges. Some good points, though the hostile tone weakens the argument.

I don't think Docker should worry too much about backwards compatibility right now, just power forward until it's got a solid formula. In the meantime, caveat emptor!

If you are running 1-2 of copies or applications that very large amount of memory while being pretty small themselves, not much (and that's admittedly pretty common).

However, if you, for example, need to run 20x copies of 50MB application that uses 100MB memory at runtime in somewhat isolated environment, you would only provision machine with 20x100MB+50MB=2050MB + whatever OS needs (~100MB?) to keep everything in memory. If you made VMs for each of them, you would need 20x100MB+20x50MB+20xOS overhead=5000MB'ish, 150% increase in this case.

Also, starting up a container is a lot faster than starting a VM. If image is in the local cache, your application likely starts loading within 500ms. VM startup times are usually measured in minutes.

I can type

    cf scale -i 100

And have 100 copies of my app running in a few seconds, depending on cache warmth. Or a thousand.

VMs booted up from images are much slower. And cloning checkpointed VMs is not apparently something AWS or GCP can do.

Disclosure: I work for Pivotal, we're one of the companies who make `cf scale` possible.

The problem with this article is it is the kind of thing I could see myself writing after about three weeks of playing with the tool, when I did not yet understand the best ways of using it... but it is written from the POV of someone who does not know what they are talking about.

Half of the article is various complaints about AUFS, and then Overlay. There are at least 6 storage drivers off the top of my head, and I feel like there are one or two more I am forgetting about. What made you choose AUFS? What were your thoughts on Overlay? Instead of going with the default and complaining, maybe research your options? Even worse though, the filesystem driver is completely beside the point. If you want to store things on the filesystem from a Docker container, you don't use the copy-on-write storage system! You create a volume, which completely bypasses the storage driver. There are even a bunch of different volume drivers for different use cases- say mounting EBS volumes automatically for your container to persist data to? His whole rant about data magically disappearing has nothing to do with the tool, but that he didn't take the time to learn how to use it correctly.

Further, while I am sure he is telling the truth and has machines randomly suffering kernel panics and crashing multiple times per day, it seems hard to believe that it is docker causing this. He is running 31 containers spread across 31 aws hosts... I am running thousands, across hundreds of aws hosts, and have been for a long time, and cannot recall a single kernel panic caused by docker.

When I first saw Docker I saw something with all the characteristics of a developer fad, namely that it was a magic new thing that in reality was nothing but a barely-novel rearrangement of old things but promised to make a lot of very old very hard problems go away.

Even worse, it was a fad that adds a whole lot of complexity to the system. Anything that adds complexity always gets a jaded glance from me. Unless there's a very clear and good reason for it, increasing complexity is typically a sign that something is wrong. (I use this rule in my own code and designs too... if I'm building a tower of Babel then I'm not seeing something or making a mistake.)

"Let's take preconfigured Linux system images and treat them like giant statically linked binaries."

Don't get me wrong-- it is a somewhat interesting hack and does have some utility. (and it IS a hack) But the problems it solves are in fact a lot harder than they seem and stem from fundamental 30+ year old issues in OS design and the Unix way of doing things. Our OSes date from the days when a "box" was a very long lived stationary thing that was provisioned and maintained manually, and they're full of things that today are bad design decisions because of this... like sprawling filesystems full of mutable data with no clear mutable/immutable division.

But the hype on this thing was just insane. I've never quite seen anything like it except maybe Java in the 1990s and that had more substance (a popular Smalltalk-ish language). For a long time I've joked that devops conference conversation was sounding like:

"Docker docker Docker docker docker docker Docker docker docker?"

"Docker docker docker."

"Docker, docker?"

"Docker docker docker Docker docker."

"Ahh, Docker."

Luckily it seems to be dying down and a lot of people are realizing it's not a silver bullet. This will probably inoculate a generation of devs and admins against over-hyped fads, but never fear... a new generation of devs and admins are coming in now and are eager to rewrite everything again since this time they'll be able to make a few trivial rearrangements of things that are already there and this time they'll get it right.

Java is the most 'popular' language in the world in 2016 (TIOBE), so maybe the hype was well placed.

> Google invented containers long before it was a thing

I had to laugh at this one, then again I am an old dog.

Solaris zones ~~

Does it count if you implement something but the UX is so terrible that no one ever uses it?

Nobody really remembers the Diamond Rio Karma but everyone knows about the iPod. History is only acknowledged by historians, perception is reality otherwise.

I think the fact that all portable mp3 players had terrible UX and were not successful until the iPod kind of proves my point here.

Before the Karma the Rio PMP300 was very cool! Portable mp3's! You could load up ~20-30 minutes on its 32mb. This completely blew away the first mp3 player I saw which could only hold 8mb which was still amazing in its own right.

The company I worked for in 1999 was using HP-UX containers, aka Virtual Vaults.

I could also add the several solutions done by mainframes, or other UNIXes like Tru64.

I dunno where you were in 2007, but every big shop was running Solaris zones at one point. And AIX LPARs before that.

I worked at a university then, no one was using solaris zones. I think they had the 2 big AIX boxes that had lpars enabled in a fixed configuration that never changed.

all solaris and aix was ripped out and replaced with linux.

Solaris zones UX is still unsurpassed to this day. I'd pay a lot of money for Linux container tooling as polished as Solaris Zones.

The problem with zones was that you used them to run solaris which as a distribution was a complete disaster. Maybe it worked if the only thing you them for was to run oracle.

I had created a bunch of solaris zones and then had the great idea to patch them.. everything broke.

I'm actually using smartos at home now. kvm and debian lx zones. works great. A solid platform with a UX that doesn't make me want to kill myself.

One man's trash another man's treasure. Solaris 10 packaging was awful, but IPS in Solaris 11 (soon 12) and OmniOS is great and the integration with zones is fantastic.

Well, you did say "Solaris zones UX is still unsurpassed"

Still implies that it was always good..

It was absolutely terrible in Solaris 10. Maybe it's better now. Maybe with solaris 11 they fixed it so you didn't have to chose between a full zone that took 5GB or a 400MB sparse zone that would break in fun ways when you tried to do anything.

At the time I was using xen+debian and a 'full' vm for me was about 200MB. Basically base system+sshd+whatever daemon I needed to run. Attempting to do the same thing using zones was a complete disaster.

Sorry, Sun doesn't get to take credit either. IBM's VM did it decades earlier.

Well, I lost nearly half the article asking myself why would somebody ever run a single container per VM... They got a bug that made container-running VMs unreliable, thus did a non-test using what they got available.

I'd be more interested on articles that actually tested it running how it was meant to. Or, if you can't, just stop there and say "there's a showstopper bug, it wouldn't work for us" - those are useful articles too, and require much less effort.

About databases, mostly anything¹ you put between them and the hardware will hurt you. VMs are bad too. What I can't imagine is how you would create and destroy database instances at will - I might live in a different reality, but around here you'll always want as few database nodes as possible, and as a consequence, as powerful and reliable nodes as possible, what means as little moving parts as possible.

1 - What won't hurt is a disk abstraction layer designed for performance and as expressive as the original physical stuff. This is widely used with the "SAN" name, and I'm willing to bet you can mount one inside a container. But well, on AWS you really won't.

With a database, the data itself is the important part- using something like Kubernetes combined with persistent volumes (GCE or EBS) lets you easily move a running database between VMs with minimal effort. You run the program however you want, and let the orchestration layer bring the persistent data to wherever that program is running.

The critical bugs left are actually race conditions. They happen rarely, more often under high load.

Everything was fine in testing, everything was fine in production... at first.

Fast forward a bit. More application put in docker, a lot more load in both testing and production. Things go wonky.

If we ran our current systems in Docker/Debian from January, a host would crash every hour and we'd have figured out the unstability. Instead the docker adoption happened slowly, while a few patches (and temporary regressions) were released along the year.

It seems I have seen a lot of posts on HN relating that Docker is not ready for production. I am using Docker in production on my small company for one year and it's working pretty well (maybe because we are still small).

Being caught in the containerization craze myself I'd love to hear whether the story is exaggerating or is painfully accurate.

So far I've been bitten by the inability to clean up images of certain age.

UPDATE: Another really annoying thing is the inability to tag an image directly into a registry (AFAIK). You need to pull, tag and push back again. Given that images can be GBs long you end up with really heavy net traffic for a simple task.

I'm confused by this.

You obviously have a process to build new images... why doesn't the same process prune old ones ?

No, there is no simple command to remove old images because the creators of docker want you to compose other existing commands to delete them and it would be unnecessary to add a command. IMO this is a terrible decision but that was apparently the rationalization.

A new "docker prune" command is added to docker 1.13 to address this, see https://github.com/docker/docker/pull/26108

That's what I do but given that the process is run in parallel being part of a complex CI system I needed to write quite a bit of logic to do the pruning safely myself. Main problem being that I couldn't rely on docker daemon not deleting images that might actually be still needed by other processes building in parallel.

The team at spotify has built this (see above): https://github.com/spotify/docker-gc

Thanks! That looks promising.

Why isn't that just built-in?

Why do you want to retag images? I'd suggest simply not doing this and adding all the tags and labels you need at build time.

One potential reason is to handle version promotion.

For instance, you may want to run an image for a while on a staging server before re-tagging it as the current stable release.

But then you could just push from the staging server. And it's probably best to keep this independent of the image tag i.e. your system knows image v1.1.2 is the latest stable, but the image name never changes. If you retag your stable image "latest", you run into issues with rollback and figuring out what you're running.

But now your staging server is also your docker image management server, breaking the JLP contract.

What do you mean by JLP contract?

We are currently moving to using Docker swarmkit (from non-Docker environment). We thought about using tags to communicate environment but it felt like an anti-pattern given how tags work. What we ended up with is the following:

Per publishable branch (e.g. master to prod, dev to QA, shared integration that isn't ready for dev to secondary QA):

- Publish image to private docker registry with tag in the format of <branch>-latest (e.g. master-latest, dev-latest). Also add labels with git revision hash and CI build number to reconcile what is in the image both in CI and in git repo.

- Capture the digest of the published image and put in .txt file as an artifact from CI.

- Add a tag to the git repo with the build number, treated as version number in the form of v<build-number>-<branch> (e.g. v1-master).

- Maintain list of applications' docker image digests that should be running in each environment. We currently have this automatically determined by pulling the digest artifact from the passing-build digest. We do allow for manual overrides, however.

- We have a CI project that updates the docker swarm with these digests and other settings (replicas, mounts, environment variables). We wrote a small tool that does this similar to where stacks/dab files are going but with more functionality.

With the labels and published digests we can go from build to digest, digest to build, etc.

This is working really well so far and we aren't fighting the tools. We may separate the non-master registry later in life, and that's really easy given that docker treats the private registry hostname as part of the tag.

Exactly

A lot of the issues in this article seem to stem from running Docker on old kernel versions.

> We are using Debian stable with backports, in production. We started running on Debian Jessie 3.16.7-ckt20-1 (released November 2015). This one suffers from a major critical bug that crashes hosts erratically (every few hours in average).

If you're stuck on an older system that's poorly supported by Docker then it may be a bad choice for you.

Is < 1 year old an old kernel version?

Honestly, I liked this article because they had been using Docker for more than just a couple weeks.

To be fair, I have not had the honor to be a docker customer. But regarding the kernel thing, I have limited IFY. Let's say if you goal is to keep yourself up-to-date with kernel tree, probably you have to compile the kernel DIY. The hardware dependencies and customized kernel config is not that handy. Then if you pick a distro, probably, not sure it is a problem, things like initd sneak into systemd/sysctl brought the element of surprises occasionally. While, to move or not to move, both are forwarding and backwarding.

Which actually stable distribution includes a sufficiently new kernel? For important stuff, a lot of us stick to Debian stable, Ubuntu LTS releases, CentOS/RHEL, etc.

CoreOS only. They're on 4.8 right now and they update the kernel whenever is necessary to keep up with Docker needs. That's the only safe bet on the long term. (as stated in the Roadmap in the article).

Read: Even if Ubuntu 16-LTS and Centos/RHEL 7 may be up to date enough now (not sure), they won't keep up with the latest minor kernel updates and that will become a problem in 6 months when docker 1.1X will require THE newest kernel.

It'll be interesting to see how this plays out. RedHat is pretty invested in Docker right now via OpenShift so they have a lot incentive to backport required kernel fixes to Centos/RHEL 7 but at some point they'll likely have to drop support for the latest and greatest Docker release.

What makes you think they aren't just stabilizing/standardizing a single version of docker and backporting fixes?

After all, about 1/2 the gripes stemmed from the fact they docker itself is basically the engineering branch minus any serious regression/etc testing. Of course things are going to break from release to release, that is why test departments exist.

Ubuntu 16.04 LTS is on 4.4, and that package is available as a backport for 14.04: https://wiki.ubuntu.com/Kernel/LTSEnablementStack

What a very strange article. TES were using docker in production in October 2014. It was certainly possible to run docker containers (docker run -d) in the background and to view them (docker ps) as early as October 2013 when we first evaluated it at the Daily Mail.

Author's company should hire a proper devops person and then docker, magically —from the devs pov— will start working.

AUFS on production? Seriously? Create a BTRFS partition. Docker isn't a shortcut to learning a linux based OS, it is a useful tool in the hands of people who know what they are doing.

To the people downvoting:

If your company lost thousands of man-hours on a tool and has nothing more to show than a blog article with half the facts coming out of the author's head and the other half boasting about bad production practices, would you prefer me to sugar coat it?

A hire that knew linux and docker, would solve the problem.

I work with docker since 2013. I came to a team with devs that already used docker. I didn't know anything about it at the time. They gave me a docker image with a JVM based app, that had a two years old JDK, ran as root and the container was ran with privileges enabled. Did I have to know docker to tell them that this was nuts? Nope, I had to know ops. For the same reason I knew AUFS before I learned about docker and I knew very well that I would trust AUFS on my NAS for media storage but not as storage for a production database.

Did I ever tell the devs how to write java? No. It isn't on my job description for a reason.

There is a dissonance here.

A linux guru will know that AUFS is unstable... but the problem is mentioned nowhere and Docker still uses it as the default filesystem for [most of] all cases.

The linux guru will know to avoid AUFS... but the replacement [overlay2] was rolled out very recently and it's only available in the latest OS and systems.

AUFS isn't included in the default setup of many Linux distributions. CoreOS, CentOS, RHEL do not have it. If I recall correctly, CentOS and RHEL will use the device-mapper driver by default and CoreOS the btrfs. I would speculate that the docker team gave so much attention to AUFS to lower the barrier to entry but it isn't the only option.

My choice is btrfs and it has been rock solid for me since the start. Of course one has to know a bit about btrfs and always leave a healthy percentage (I'd say 10% or more) of the filesystem free.

I think AUFS and overlay2 do have an advantage of their own though (apart the works-out-of-the-box). Due to the way they work, you can have shared memory between containers. Thus if I had to run many containers with a JVM app, I would give them a try to lower the tax on RAM.

I think you're confirming the point of the article.

Gotta find the right combination of OS & filesystem to have a stable docker.

> If I recall correctly, CentOS and RHEL will use the device-mapper driver by default and CoreOS the btrfs

CentOS & RHEL uses device-mapper with loopback device by default (and logs "Usage of loopback devices is strongly discouraged for production use. Either use --storage-opt dm.thinpooldev or use --storage-opt dm.no_warn_on_loop_devices=true to suppress this warning". Also related article [1]

CoreOS uses overlayfs by default now from a while [2]

[1] http://www.projectatomic.io/blog/2015/06/notes-on-fedora-cen...

[2] https://lwn.net/Articles/627232/

Thanks for the correction. It has been a long time since I had to setup docker manually.

> A hire that knew linux and docker, would solve the problem.

I'd pick a hire who also knew a platform. I like Cloud Foundry because I work on it, Red Hat folk would prefer OpenShift, there's also Deis and Convox and others I am cruelly neglecting.

Containers are a building block. An important one, they make new architectures practicable. But past a certain point, rolling a custom PaaS that you will have to maintain at your own expense, with no commercial or opensource support whatsoever, forever, doesn't make much sense.

A hire that knew, in 2013, that BTRFS was a good bet and AUFS wasn't? BTRFS still isn't a good bet, and certainly wasn't then.

The author made some really good points, but CoreOS is made by Docker for Docker is not one of them.

CoreOS is the company behind a competing container tech called rkt (rocket) and CoreOS Linux.

I have to disagree on the pain of working with Kubernetes. There are definitely gaps in documentation, no doubt about that.

But it took me under a day to containerize an application I'm working on, and get it up and running flawlessly on Google's container engine. And this is coming from having zero Docker experience.

Not as fast as you, but I requested myself for about 2-3 weeks and went from 0 Docker/k8s experience to having a modern SOA rails app up and running on a GKE cluster. Are you using some type of CI/CD tool to deploy? How did you setup DNS?

I didn't get that in-depth, which is why I was able to do it so quickly :) I just deployed to Google Container Engine using some docker images I built to replicate a rethinkdb/node app we've been working on. I used DNS for service discovery in the app, and didn't hook into a CD tool. Just used the google command line tools.

>Making a new filesystem in 1 year is still an impossible mission. Docker just tried and failed. Yet they’re trying again! We’ll see how it turns out in a few years.

Is this just fear mongering or are there any actual issues with overlay2. we have been trying out overlay2 on our less critical systems and haven't experienced any issues so far but we have stressed it enough to know for sure.

  That was a 7 hours interplanetary outage because of Docker. All
  that’s left from the outage is a few messages on a GitHub
  issue. There was no postmortem. It had little (none?) tech news
  or press coverage, in spite of the catastrophic failure.

This is incorrect. There were github issues, HN posts, and so on. Solomon Hykes posted on Github [0] and HN [1]:

  Hi everyone. I work at Docker.

  First, my apologies for the outage. I consider our package
  infrastructure as critical infrastructure, both for the free
  and commercial versions of Docker. It's true that we offer
  better support for the commercial version (it's one if its
  features), but that should not apply to fundamental things like
  being able to download your packages.

  The team is working on the issue and will continue to give
  updates here. We are taking this seriously.

  Some of you pointed out that the response time and use of
  communication channels seem inadequate, for example the
  @dockerststus bot has not mentioned the issue when it was
  detected. I share the opinion but I don't know the full story
  yet; the post-mortem will tell us for sure what went wrong. At
  the moment the team is focusing on fixing the issue and I don't
  want to distract them from that.

  Once the post-mortem identifies what went wrong, we will take
  appropriate corrective action. I suspect part of it will be
  better coordination between core engineers and infrastructure
  engineers (2 distinct groups within Docker).

  Thanks and sorry again for the inconvenience.

[0] https://github.com/docker/docker/issues/23203#issuecomment-2...

[1] https://news.ycombinator.com/item?id=11823231

This is not a postmortem; this is a github comment copy-and-pasted to HN, saying "sorry" and talking about a future postmortem.

So what did go wrong?

As said, they pushed packages with invalid signatures.

Why did this happen? There wasn't any explanation given. I'd love to see a postmortem from Docker.

Exactly.

We've been using docker in production for two years now. Started with docker 1.4 and Rancher v0.15. Docker has had it problems with backwards compatibility, but we haven't had the mentioned filesystem problems.

Our containers are completely stateless. We use AWS's RDS and S3 to store state.

Since OP is running on AWS anyway, why not just use Amazon ECS? The orchestration (including Elastic Beanstalk) is pretty good, and we haven't had any stability issues in production with it. Same thing for registries; just use AWS ECR instead.

Here here. I operate hundreds of ECS clusters and it's great.

> The impossible challenge with Docker is to come with a working combination of kernel + distribution + docker version + filesystem.

Amazon has met this challenge with the ECS Optimized AMI. It works, gets frequent upstream updates, and you can open tickets against it and get great support.

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/e...

* Amazon Linux

* Linux Kernel 4.4.23-31.54.amzn1.x86_64

* Docker version 1.11.2, build b9f10c9/1.11.2

* Device mapper

I also have been steeped in Dockerland for the last couple of years. Here are my own Docker pros/cons. Caveat: I am not nearly the power user that thehftguy is or understand Linux kernel issues in any real depth. I’m a lowly high-level language programmer (elisp/Python/Clojure/Java/shell).

Pros:

When Docker works, it’s great. The concept of being able to install software on the users behalf in the container and have the user install the container really does make life easier. The unwieldy command lines are nicely abstracted with docker-compose(.yml). Software upgrades via containers becomes a nearly trivial process.

I work on scientific software that can be a huge pain to install and configuring the packages correctly often times requires the help of the software creator. The fact that you can codify this “dark knowledge” in a Dockerfile is of tremendous benefit. And generally speaking, I like the concept of the Dockerfile. Whatever container technology ends up winning, I hope there will always be something like the Dockerfile.

Cons:

- Docker version upgrades never go smoothly. Firing up docker after apt-get update && apt-get upgrade never works with respect to Docker. You get some obscure error and after spending 30 minutes in Google you end up having to rm -rf /var/lib/docker/aufs. My solution is sometimes to throwaway the VM and start over, but this is unacceptable.

- File ownership issues inside versus outside the container are (or at least used to be) a huge pain in the ass! I am referring to files mounted from the Docker host file system into the container. (And I am also referring to running docker as unprivileged user that is part of the docker group.) The solution I settled on is to enter the container as root, run chmod/chown on the appropriate directories and drop down to a “regular” user with gosu. At that point, you proceed on your merry way by starting your service, process, etc. This work flow solves permission problems inside the container, but outside the container is a different story. At least on earlier versions of Docker, you could lose permissions of files created via the container because at that point they are owned by root as seen from outside the container! More recent versions of Docker seem to solve this problem as I have gleaned empirically and from experimentation but I have yet to see adequate documentation about any of this and I have spent a long time searching! With more recent versions of docker I have observed that the files in question are no longer owned by root (again as viewed outside the container), but by the user that started the container, the behavior I would expect. I’m also not crazy about the gosu solution which seems like a bolted on rather than baked in solution.

> My first encounter with docker goes back to early 2015. Docker was experimented with to find out whether it could benefit us. At the time it wasn’t possible to run a container [in the background] and there wasn’t any command to see what was running, debug or ssh into the container.

Having worked all of 2015 in the container space, I can state this comment is a false statement. While it may not have been possible for this person, the technology has existed for quite some time to see what process are running in a container, debugging became possible with sysdig and ssh'ing into a container has been a thing for YEARS.

Excuse my ignorance on docker, but why did the OP not use a proxy cache for the registry? [0]

If they were running this on a single machine with backups they could restore to a time before the incident.

Also querying and downloading from the public registry everytime time seems like a waste of bandwidth.

[0] https://blog.docker.com/2015/10/registry-proxy-cache-docker-...

Anyone who has used docker at all knows the above statement is 200% rubbish. FFS

Yeah, it is! I was using it pre-2014 and docker top was available then and the standard way to debug was to set up an ssh server in the container. You only had to google it.

Or just use lxc-attach

We're successfully running Dokku instances for various dev/qa environments, but I have not attempted containers in production as of yet.

While Kubernetes seems to be the orchestration choice du jour, I had been eyeing http://deis.io/ (which I believe to be at least as mature). Can anyone comment on their experience or thoughts on Deis vs Kubernetes? TIA.

Deis is standard kubernetes + deis addons like their workflow.

Deis is built on top of Kubernetes.

Ok, I see that now, it was not always the case...it has been awhile since I looked through their documentation. TY!

Yep, we made the shift over from Fleet (a CoreOS project) to Kubernetes in v2. We were just evaluating Kubernetes at that time.

> Making a new filesystem in 1 year is an impossible mission

If this is "real" filesystem for disks, not proxy for other filesystems, mostly read-only.

Over the course of the years I've tried to like Docker. I've found the benefits to be not worth the stress and aggravation for the associated "magic" that it takes to provide the kind of user (developer?) experience they're after but have thus far failed to achieve. It's like the uncanny valley of sysadmin tools.

The concept of getting away from snowflake machines and making your infrastructure more code-like with deployment scripts is the way to go, but you certainly don't have to use docker for that. At the end of the day, it's more about forcing your team to be regimented in their processes and having a working CI system for your systems related stuff. You should be able to start from nothing and with a single command have a developer's environment up, all the way to a fully scaled prod system.

Docker is just a tool to try and get to that point of evolution (albeit a poor one).

After reading your post I think that you miss understood the purpose of Docker.

Even if people try to sell it has flexible silver bullet, made up for everything; like any other tools around, they need to apply some sort of CAP theorem. Because underwoods we're just talking in how moving bits.

For instance I use containers like AWS lambda's functions. And I use tar streams to pass the contents inside them (So no virtualised mounted filesystem needed).

With that I mean you can find different way to use them, in short: Law of the instrument.

If you're wise (and according with you article you got ton of experience), you can find good purpose for it.

Docker is just 'meant' to make your life 'easier', like any other tool, not always they satisfy the expectations.

In other hands you gain experience with it, I'm sure you would use them back for a better purpose, where they can perform better for you, there is nothing to worry about this is just experience.

Thanks for sharing.

Interesting! A relevant topic ("Hyper is Docker Done the Right Way" by TheNewStack) came up in HN first page as well.

Is it how Docker should have been designed to embrace production in the first place?

https://news.ycombinator.com/item?id=12873089

I've been using Docker in production for over a year at this point. I can't say that it's been problem free, but nothing so dramatic as this post outlines. For example, I run Erlang apps in it, with heavy production use.

Heck, it is possible even remove images from the docker repository without corrupting it, just not from a single API endpoint. FWIW I think that it is a big failing of the v2 repository project to not have that functionality from the get go.

It's also worth noting that I've only run Docker in production using CoreOS and Amazon's ECS AMI. Both have their drawbacks, but nothing so dramatic as to keep me from recommending Docker in production for "cattle" style applications.

The section on not running databases with Docker isn’t really fair. Of course you wouldn’t use something that crashed all the time to run your database server, but Docker’s not designed to crash. It has volumes and you can do regular crash recovery with it. Crashing hard and taking the host’s data with it shouldn’t be a likely scenario and should be prepared for about the same as disk failure – have redundancy and make backups.

From the article itself:

> Containers are not meant to store data. Actually, they are meant by design to NOT store data. Any attempt to go against this philosophy is bound to disaster.

Yes, I am replying to that point. It’s incorrect, or at least misleading. Containers are meant to store any data that they have to externally; volumes are a facility to do this. Storing your data in an ephemeral container would absolutely be stupid, and the risk of unrecoverable corruption after a crash should be low if you’re not experiencing frequent kernel panics.

> The impossible challenge with Docker is to come with a working combination of kernel + distribution + docker version + filesystem. Right now. We don’t know of ANY combination that is stable (Maybe there isn’t any?).

You can do worse than use Red Hat Enterprise Linux 7 (RHEL7) that has a tested and supported blend of those components. Or CentOS if you just want to try it out and don't care about support.

Docker is unstable trash and does not belong anywhere near a production environment.

Unsurprisingly the linked article is fraught with glaring errors and misinformation. This has become typical of most things I read that is docker-related; it's a circus.

It saddens me to see the Linux community get dragged into and overwhelmed by this mess.

It's funny how the article mentions AWS lock-in but fails to recognize the infrastructure lock-in aspect of relying on a whole container/orchestration/virtualization stack that is so fragile that (according to the article) the only way to reasonably use it in production is to outsource its maintanenace.

Once you go down this path, it seems there is a very a limited number of providers capable of providing the infrastructure you require (at least compared to vps/dedicated/datacenter providers) and they can just keep pushing you into using more and more fragile infrastructure components that you would never want to maintain yourself.

Eventually, you may very well find yourself paying so much for all the hardware/virtualization/os/container/orchestration maintenance and resource overhead, with a number of backup instances of everything due to the unreliability of all these components, that you wish you could just go back to placing a few pieces of metal in a few datacenters and call it a day.

I feel like almost all issues in this article stem from not following ops best practices. And similar issues would happen no matter what technology was being used, they just happen to blame their issues on docker because that's the tool they were using at the time.

Clean docker images:

  docker rmi $(docker images -f dangling=true -q --no-trunc)

Docker 1.13 is shipping with cleanup commands built in.

Just saw you comment and googled for details: https://github.com/docker/docker/pull/26108

Dangling images are just those that are not tagged. That happens if you create a new image and tag it with an already existing tag (e.g. latest).

This command won't help you clean up disk space if you build/pull images each with their own unique tags.

Meanwhile BSD jails & Illumos zones are humming along nicely on ZFS.

Does anyone know is rkt much better in terms backward compatibility?

I think Docker is meant for apps in a shared environment (with other apps). If you run dedicated app servers Docker is an unnecessary abstraction.

I've been running docker in production for about 3 years.

Almost every time I've hit a problem, it has had nothing to do with docker (though everyone likes to blame it). 99% of the time the issues I've encountered are due to docker configuration (and yes the learning curve is steep) or the app itself crashing.

The one time I've encountered an issue with docker with databases (used elasticsearch) was a kernal issue (where the host kernal was out of date). That was the only time I had to care about the host OS in debugging a docker issue.

One thing that I recommend for anyone getting into docker - use docker-compose for everything. Don't use the docker CLI.

One of the major problems I have with docker is that (especially for beginniners) the docker command line tool can cause unintended consequences.

Every time you run docker run <image> you are technically creating a new container from an image. Once that finishes, the container still exists on disk. Using the --rm flag will cause the container to actually be removed after running it. I sort of wish docker would emphasize that flag, because often when developers are debugging or trying things, they will continue to run docker run <image> multiple times - not realizing that they are creating a ton of containers.

Yes, the containers are usually very light, but it caused a lot of confusion for me when I started out. And part of my superstitious self believes that the less dead containers you have the less likely you'll have filesystem issues.

A second source of confusion for new devs I've noticed is that the concept of volumes are somewhat confusing. "Why would I ever do COPY in my docker file if I can just volume it?"

Once you've worked with Docker, like any tool, you become more adept at avoiding a lot of the mistakes mentioned in this article. Which leads me to a point I can't disagree with the OP about - upgrading docker is annoying as hell.

Almost everytime I upgrade docker I end up having some obscure error. When I look on docker github I notice I'm not the only one, and often times the issues are just a few hours old.

But, what the OP didn't seem to realize, is that to avoid these issues you need to lock your docker version in production to one that you know works. Additionally, you need to not build your dockerfile in production. Build your docker containers with a build system and upload them as artifacts. You can either use an internal docker hub or do docker save to save them as a .tar.

Building while in production is a real no-no, even though it may seem more attractive than moving around 300-500MB images. You never know if that one dependency you apt-get in your dockerfile is offline for some reason. And you can't always depend on devs of dockerfiles properly versioning them anyway.

the --rm flag only removes the container, not the image. That doesn't help.

Ha, docker wishes that they had the staff to create all the things this article claims they create.

Is it just me or this article mostly shows serious incompetence from the author/company?

If not Docker, then what?

Well there's LXC, LXD and rkt to begin. Docker have themselves helpfully begun extracting libraries.

The Garden container API which underpins Concourse CI and Cloud Foundry has switched to runC as the container runtime. So far works fine, with no Docker daemon required.

Disclosure: I work for Pivotal, we sponsor Concourse and donate the majority of engineering to Cloud Foundry.

Docker has been the go to solution all my python-for-web-development friends have been using as flagship to "kill php et al" and sharing this with every single one of them is making smile again. Thank you OP.

That such a bad article ends up here shows the biggest problem with Docker: It's irrational hype. Whether it leads you to hate it or want to dockerize all the things, it's unhealthy.

Sensationalism and hyperbole sells. We all know that. We also know that the truth almost always lies somewhere in the middle. The writer of the post clearly bears a lot of scars. Self-inflicted? Maybe. Avoidable? Possible. Is Docker at fault in some ways? Definitely. In any case, such articles still serve to balance all the other hype and remind us where we're at on the Hype Cycle curve with respect to Docker containerization. ;)

Let's not forget to LOL about those 2 animated GIFs. Just about ejected my morning coffee through my nostril.

Docker SHALL NOT run databases in production, by design."

You need to use external volumes

I have some sympathy for the OP here, as I have heard of a lot of people running into pain with DBs. But yes, they definitely should not be writing to the UFS, as the OP seems to be doing. However, even if you're using volumes, you need to be careful about what happens when images change etc. It's not trivial.

Wait, do you think he's not using data volumes?

I was assuming he was complaining about them even with data volumes.

I recommend he try a better solution such as tredly.com, or perhaps SmartOS. Some things actually work OP, don't give up hope because of docker.

I'm not getting in to this discussion... nevertheless, this guy just made my day!

This was a fun read :)

Automatically updating core infra from public repositories without CI'ing.....

No... losing a day of development is on their infra team.. not docker.

Can we please, please, please get a follow-up titled: "Bash in Production: A History of Failure"? With a comedic yet tragic theme of course.

Docker in Production: How we did it wrong

This is a typical case of blaming the tool. There are so many bad practices mentioned I don't know where to start. No orchestration... One container a host... Not using volumes correctly... Apt-get installing docker every time ci runs... Not using an optimized base os for containers... It's amazing what people will blame their problems on when they don't do their due dilligence with a new platform.

> One container a host...

As said in the post, this was a consequence of Docker failing them when running multiple containers on a host.

> Not using an optimized base os for containers...

On a server that might be possible, but I thought Docker's advantage was providing a reproducible system that can also be run on your dev machine? Sorry, if Docker doesn't run stable on my dev machine, no way I'm trying it.

The topic is running containers in production. Why not use a base os designed for that?

Because the base OS needs to be something an admin can work comfortably on, esp. if the host is supposed to run other things besides docker (e.g. a database, you do not want to run mysql over a network if possible).

And for this you need a "real" base OS, which will be either apt-flavor (debian/ubuntu) or rpm-flavor (rhel, sles), depending on the organization.

Why not run your database on a different machine then. Separation of concerns. Have a cluster a machines for containers so you can take advantage of it's strengths instead of trying to duct tape everything together on machine

> Why not run your database on a different machine then. Separation of concerns.

While true in general, you always end up introducing latency (simply because it's on a different host) and possible packet losses/retransmits (e.g. when the switch buffer memory overruns). Given the "hft" in the blog name (high-speed financial transactions), both can equal losses of money.

Is 'An History' in the title some sort of meme joke like 'An Hero' [1] or does the author not know basic grammar?

[1] http://knowyourmeme.com/memes/an-hero

No it's not a joke and yes it's valid grammar. If you're going to correct someone's grammar, at least know what you're talking about.

https://www.quora.com/Which-one-is-correct-a-history-or-an-h...

Hmm, if anything that link seems to imply it's invalid - plus it's Quora, not exactly a high quality reference.

The rule is "an" is used in front of a word that starts with a vowel sound[1]. "An hour" is correct because the "h" is silent. All modern (native) English dialects pronounce the "h" in history, ergo "an history" is incorrect grammar.

[1] https://en.wikipedia.org/wiki/English_articles#Distinction_b...

"In British English, either is acceptable"

from the first sentence.

> Quora

> not a high quality reference

> proceeds to reference Wikipedia

What's your problem with Wikipedia?

none but I wouldn't call it a more reliable source than quora for something like this.

"In linguistics, elision or deletion is the omission of one or more sounds (such as a vowel, a consonant, or a whole syllable) in a word or phrase."

Source: https://en.wikipedia.org/wiki/Elision

Well, the grammar rule got a name. It doesn't say whether it applies to "a[n]" history" though.

It's based on the vowel sound. The author may be someone who drops the h when they pronounce "historic". See: http://www.betterwritingskills.com/tip-w005.html

Author is probably French or something.

critiquing grammar on the internet is generally a bad idea these days because:

1. typos

2. not everyone is a native anglophone. the web is global.

3. nobody is an actual grammar expert, including you.

4. avoiding actual discussion is obvious

3b. supposed authorities are in complete disagreement on fundamental points of grammar (eg. oxford/cambridge disagree on has/have in third person verb of possession; oxford is right historically, cambridge teach far more 2nd language speakers)

Reply on news.ycombinator.com