Comment by nvarsj | Hacker News Reader

Comment by nvarsj | Hacker News Reader

Parent Full thread

Docker is a poorly engineered and over-hyped technology.

The concept is great - and in fact, many companies have built great tooling around Linux cgroups. It lets you efficiently binpack applications on a single server - which is why 'containers' were created in the first place.

The side benefit of letting you define your OS libraries, and other things, is a nice bonus, and way overblown in my opinion.

Docker and its tooling is just plain bad. It's so bloody unstable, unless you pick some esoteric combination of versions and storage backends (old docker + old aufs + old ubuntu seems to do the trick) - and even then, you'll run into problems. Documentation won't help you here - it's trial and error.

The docker group seems focused on feature, feature, feature, while the basic stability and performance remains poor. It's quite amazing how they get it so wrong. Browse through the github issues for performance and hanging issues. The surrounding tooling is poor as well - registry v2 doesn't even support deletes (because they need to maintain compatibility with the 100000 different storage backends for it). There is no LTS release, bug fixes only happen in latest version, so you're in a constant state of brokenness. And so on.

We used Docker sort of early on, and got out of it around v1.6. Many problems that you had to work around yourself, but the one you just reminded me of was the repos: There was Dockerhub, a third-party place without 100%(-ish) uptime, and two serve-your-own, one of which was a black box you could never delete from, and the other had stamped "NOT FOR PRODUCTION USE" in big letters on its github page.

I remember one of the engineers giving a talk, saying that the problem they had was growing too big too quickly - they didn't have time to properly work out the base architecture in the early days.

I think the real problem with Docker is Docker Inc. The pressures and constraints the company is under promotes the creation of new features that can be marketed and new software that competes with products from the competition.

They have effectively no incentive on getting rid of bugs in the core product or to test features they do add exhaustively.

Sooner or later someone actually using containers will produce a docker replacement that will take over unless Docker focusses on what actually matters.

Docker Inc. desperately wants to be VMware. I don't think it's a coincidence that the official documentation is so full of virtualization-like choice of words you'd be forgiven if you thought they already were.

They do however seem to be cutting so many corners technically, that the risk is they get undercut on their core offering.

Deletion from registry v2 is definitely supported now. It's just a total PITA, and took them a while to implement. But I got my disc space back, so I'm not complaining. :P

Any pointers?

This answer on SO is more or less the approach I used:

http://stackoverflow.com/a/37716286/308278

Ok, throw Docker away.

Did anybody have better experience with e.g. Rkt?

I really like: daemontools + static binary + setuidgid and maybe chroot.

Or mesos where everything needs to be in a tarball, which is extracted and sole program run.

If stuff is statically linked and related files (config, assets, ...) are part of bundle that can be chrooted, what is the value add of a container?

Containers are easier for folks with less operational experience to understand. Or maybe easier to get started with is a better way to put it. It's easy to underestimate easy to use tooling when you aren't the target demographic.

It's not hard to use control groups in daemontools family style, either.

* http://jdebp.eu./Softwares/nosh/guide/move-to-control-group....

The concept is great but it's also not original. It's called "processes". Docker is little more than a mass of complication laid atop fork+exec.

That's why nobody can get it right - because we already did.

I totally agree. The real issue is dynamic libraries and how hard it is to compile C/C++ code statically with GCC.

If you could just pass `-static` to gcc and it actually worked like you expect this would never have happened.

Fortunately that seems to be changing somewhat. Go is totally static, and Rust can easily be made totally static using muscl. You can even do totally static C/C++ apps fairly easily with muscl.

Dynamic libraries (in the C/C++ sense) only scratch the surface. Containers give you your own file system namespace (among other namespaces), which means all of the files that make up your complicated application unit can be put together and work together in isolation, separate from the machine's main file system.

Isn't that chroot jails?

Yes, quite literally, in fact: https://github.com/opencontainers/runc/blob/ee992e5ff7143ea3...

What advantages do file system namespaces have over separating by directories and users?

A big one is managing software which you didn't write: if you have two things which expect to be able to write to /etc/mydaemon.conf etc. you either need to burn a VM for each one, fork the startup scripts or take the Debian-style approach of maintaining patches which make everything configurable, or manage something like maintaining chroots directory hierarchies.

(repeat for network namespacing: it's really nice not to need to play games to have your CI server start 3 running jobs which all think they're listening on port 80)

None of that is impossible – in the case of chroot there's many years of precedent - but if you do it regularly, there's a strong appeal to automating a common pattern.

This is especially true when your goal is supporting development teams: with something like Docker, normal users don't need root just to start a daemon on a privileged port or write to a couple of files. If you work in a large or security-conscious environment, that's a fairly big draw.

(Not saying that Docker is perfect or necessarily the long-term winner in this space, only that there's a usability gap which a lot of people fall into).

> This is especially true when your goal is supporting development teams: with something like Docker, normal users don't need root just to start a daemon on a privileged port or write to a couple of files. If you work in a large or security-conscious environment, that's a fairly big draw.

On the contrary, any user that can run arbitrary containers (such as rootplease[0], for example) has root-level privileges on the host system.

[0] https://hub.docker.com/r/chrisfosterelli/rootplease/

What I was thinking about wasn't protecting against outright malice but rather mistakes and errors: If you give developers sudo access and you don't have an extremely diligent team with strong system administration experience, you're going to run into problems where people made incompletely documented changes or cause problems while working which aren't caught early enough – ever see someone break out sudo or chmod 777 as their first debugging step or even write that into the install process because it was too much work to do it right? Docker is an enormous win here both because it sharply reduces the number of times someone needs privilege escalation and because it ensures that the end result of their work can be reliably audited and repeatedly deployed.

It's true that Docker doesn't protect against compromised or malicious users with privileged access. That's a very hard problem in general which can only partially be addressed at this level — especially since many of the most damaging attacks don't need it (“The bad news: they exfiltrated our customer database. Good news: they didn't get root on the EC2 instance”). I think most of the answers for this problem are going to continue to rely on existing practices like code review, auditing, getting finely-grained SELinux / seccomp rules into the development mainstream, etc.

> If you give developers sudo access

Why would you, though? A developer would at the very most require the application's privileges, not the super user's. And that's only really necessary when doing live troubleshooting.

You have to if your developers are installing packages, working on deployment, running anything which runs on a privileged port, use tools like systemtap, etc. In some cases that can be avoided with configuration but then you're asking for bugs due to discrepancies between the environments.

You can reduce the number of things which hit that friction in a number of ways: having easily repeatable builds so a developer can test on their own VM; using a cloud service so test VMs are disposable and never shared; setting up a platform / microservices approach so a wider range of things either don't need to be touched or can be deployed without privileged or direct access, etc.

Containers (really namespaces) one way to hit that last goal: as a classic example, if you have a web app running on port 80 everyone who deploys will need to periodically restart Apache/nginx/Varnish/etc. and that may include elevated access to debug processes running as a different user. This is by far the simplest problem in this class and there are various ways (proxies, site users, firewall NAT rules, moving config into .htacccess or its equivalent, etc.) to reduce the number of times you have to care about it but but it still requires work to maintain and adapt code (especially with third-party apps).

Some people quite reasonably prefer to solve this entire class of problems by tossing everything into a container so it can run as if it's the only thing on the box.

Basically if you're a shit sysadmin, you don't have to bother learning anything or working hard.

This is absolutely fine if you intend to remain a shit sysadmin. Go nuts with Docker. The 21st century will be waiting for you when you're done.

So will 19 fucking 60 because nothing's fucking changed.

How is "your own network, your own view of the file system, your own view of the process table, your own view of the user IDs, ..." the same as "processes"?

On modern Linux distros every process is running in a cgroup and namespace by default. So these days the main difference between a "container" and a regular process is that regular processes are all jumbled together in the same root namespace, and containers are in separate namespaces.

Which Linux distributions do this? I'd like to read how "most" instances of fork and exec end up with unique networks and file system namespaces.

I didn't say unique, I said "jumbled together".

Now as far as which distros put processes in a namespace and cgroup by default, I know at least CentOS 7 and Ubuntu 15 do this. And those two distros on their own would qualify for "most".

To check if your distro does it, one way of checking is just doing a `cat /proc/1/cgroup`. This will show you what cgroups process 1 is in. By default you will be in the "root" cgroup.

To check your namespaces, `ls -l /proc/1/ns/`. You'll see the process is in some randomly generated namespace ID per item.

I'm sure you could recompile your kernel to disable this behavior, but the default reality of modern Linux is that everything is already running in a "container".

Now the question is whether or not people want to take advantage of that reality, and separate out processes in isolation, or keep running everything on a system in such a way that any single process can impact the whole system.

Plan 9 is knocking on the door and would like to have a word …

The context of this discussion would like to have a word too...

I think that Plan 9's filesystem-based namespacing (and lack of a superuser) actually have a lot to offer for container-like solutions. Any Plan 9 user can set up namespacing of the network and of resources and spawn a process within that restricted namespace.

The whole process is much simpler, I think, than that of creating a Linux container (that's the whole reason Docker exists: to simplify & abstract something which isn't really inherently complex, but is accidentally complex).

Plan 9 certainly wasn't perfect, but it had some really high-quality ideas we still haven't assimilated in mainstream platforms.

I'm not disputing any of this. But it really has nothing to do with the context of this thread...

Wouldn't it be more akin to jails?

Reply on news.ycombinator.com