Containerd – An open and reliable container runtime, by Docker | Hacker News Reader

Containerd – An open and reliable container runtime, by Docker

github.com

190 pointse-sushi35 comments

To my eyes, this signals that Docker hears the complaints from systems like Kubernetes that want to build on top of docker, but don't want all the Docker platform stuff.

Layering matters, and Docker seems to be offering a solution that is Docker compatible but more usable by layered systems.

Devil's in the details, of course, but this seems like what our customers want - docker but not Docker.

Disclosure: Kubernetes dev

As a Kubernetes user who's been running 500+ containers in production for a little over a year now, my chief complaint is not the Docker platform stuff, it's the reliability of Docker itself. We've suffered through a number of bugs that cause node instability and lock-ups and some of the underlying issues have been open for close to a year.

I don't think that this is the exact bug for our issue but it's closely related:

https://github.com/docker/docker/issues/10355

I really want to give rkt a shot but our tooling investment in Docker is huge. I want to wait and see what the runtime ecosystem looks like in another 6 months before considering a switch.

There are already ongoing efforts that are trying to get rid of that dependency too:

- https://github.com/kelseyhightower/cri-o-tutorial

- Mesos already has something called UCR that allows one to run docker images minus the dependency on the docker runtime. (http://mesos.apache.org/documentation/latest/mesos-container...)

Disclaimer: I work at Mesosphere on Mesos.

Thanks for the link, that's a pretty neat project.

I'm not a huge fan of Docker's runtime because it tends to be rather sluggish. I haven't used it in quite a while, though.

To echo this we have run 10k+ containers in production for close to a year, and docker crashes all the time. Sometimes in ways undetectable by any reasonable health polling mechanism messing up Kube as well.

Edit: Relevant details https://news.ycombinator.com/item?id=13110897

Note these were not issues with using docker poorly, or out of mainline use cases. They are all issues with dockers core networking or runtime stack that are acknowledged bugs.

For #22502, I'm looking to get in a ring-buffer implementation for log drivers: https://github.com/docker/docker/pull/28762 The same could also be done for attached clients.

#28889 is resolved in 1.12.4

#26492 was found to be not a Docker issue, but if you are still having an issue here, please open a new one, thanks!

We have a workaround in place for 26492 (a program that just queries all containers, and there VETH info and checks if they are in the bridge, force adding them if they are not) luckily so we don't have to build custom versions of Systemd. We also instituted a similar fix for 28518 (we just kill the broken container when it's detected).

However, our integration suite repeatedly caught deadlocks and some panics in 1.12.* so we have yet to upgraded to that. I'll ask our infrastructure teams to do a custom CoreOS with 1.12.4 and run it through the paces; though, I know we've had issues with Kube in the CoreOS alpha channel so it may be a no go.

Thanks for the reply!

That issue is a kernel issue in old Ubuntu kernels. If you are experiencing that, upgrade your kernel. It is not something Docker can fix. If that is not your issue, please open a new one. If your system locks up, that is a kernel issue, and as the container runtimes all use pretty much the same kernel features, switching will not fix it.

Recent versions (1.11+) have had various race conditions around stdio handling. 1.12.4 (fingers-crossed) should resolves all the issues we've come up to at this point (hopefully that's it).

The rest of the lockup issues have been related to kernel bugs which would also be observed with other tools. One major issue we've had (which is really multiple kernel issues with multiple causes) is netlink (a kernel interface) not responding and the container's mutex is held forever while we wait on netlink. The locking up part should be resolved in 1.12.4 for this as well, where it uses a timeout on the netlink socket... still going to have errors from this since if netlink isn't responding there's something else weird happening, but at least the container mutex can be released you can interact with the daemon.

In 1.14 we should have a lock-free container object so that any new issues that come up in this regard are isolated to a particular thread. This makes detecting that there is an issue harder, but something we could potentially track and even report on.

note that the daemon isn't really locking up fully, just that any command that tries to lock the container's mutex will get stuck, which includes listing it in `docker ps`, start/stop/restart/etc on that particular container.

The netlink not responding issue sounds like the one we've been battling. Now we just have to wait for a CoreOS release with 1.12.4.

I see it largely as a response to rkt, which again was largely a response to Docker either not wanting to, or being slow at, doing this kind of separation.

Avoiding thing Docker daemon, for example, was a big motivation with rkt.

But of course it's gotten even more important as Docker has ballooned in complexity.

I suspect it's more likely an immediate response to https://github.com/kubernetes-incubator/cri-o

People still want to use runc because it's a great piece of battle tested kit but don't want the kitchen sink and projects like CRI-O and Garden-Runc/Guardian are pushing the much simpler, smaller surface area angle.

Yes, I agree runc is a solid piece of kit. The rkt team is starting to use runc inside as well and hope to enable that in forthcoming releases to stay close to the Open Container Initiative specs as they move to 1.0. You can read more about that on our recent update post[1] or see a demo from a recent meeting[2].

Overall I appreciate all of the discussion on architecture of these systems and nailing this stuff down into the right format. My end goal is to ensure people get a strong reliable system, the exact technologies and projects used to get there is less important.

[1] https://coreos.com/blog/rkt-and-kubernetes.html [2] https://www.youtube.com/watch?v=x84Nes2P3Z4&t=124s

IF this announcement is all it seems, and I am expecting CoreOS folks to find any real trouble spots, this makes the various options at the lower-part of the stack much more similar.

OCI and runc were both largely responses to rkt as well, so that is pretty much variations over the same theme.

If that's what you are looking for, why not use rkt?

(Perspective: Docker in production. Would rather bite my arm off than do it again. rkt looks like a sane Docker, but I haven't used it in anger.)

Absolutely agree with this. Now I need to figure out for myself how this fits in with or contrasts with runc.

containerd is built on top of runc. Optionally you can swap out runc and replace it with any oci-compatible backend, like for exame runv.

Docker has been running on top of containerd since 1.11. This isn't really news.

Previous discussions:

https://news.ycombinator.com/item?id=11492736

https://news.ycombinator.com/item?id=10754316

https://news.ycombinator.com/item?id=13176895

I think this is the actual news: https://www.docker.com/docker-news-and-press/docker-extracts...

> Docker today announced that it is spinning out containerd, a core component of Docker Engine, its industry-leading container platform, and donating it to a new community project.

Press release linked to from: https://blog.docker.com/2016/12/containerd-core-runtime-comp...

Also containerd is going to have more stuff added - image management from the OCI image spec for example. Not a lot of stuff, but all the key runtime parts, see the Scope section for what is in and out.

yeah, this is key. It looks like JUST enough to be useful without the rest of the Docker engine.

containerd 0.2.4 which is used in Docker 1.12 covers only container execution. This announcement about containerd 1.0 roadmap extends the scope of containerd to make it a full container runtime designed to be embedded in higher level systems.

* scope table: https://github.com/docker/containerd#scope

* roadmap: https://github.com/docker/containerd/blob/master/ROADMAP.md

* new API: https://github.com/docker/containerd/tree/master/api

I love how amazingly this name ends in nerd. Ok, I'll show my way out...

lol

This was all I could see too

I think this is a smart move by Docker. The experience of docker-compose for local development is sublime. The option to deploy that local stack to any number of schedulers but with the same core container runtime feels like the right kind of prod-dev parity. Choice being the main point I'm making.

Can anyone contrast this new (rebooted?) containerd to rkt?

A "new" sytemd for containers? #sarcasm

Disclosure: Red Hat Evangelist.

Tbh, this is what would let you better use systemd to manage your containers. It's a container runtime. Not an init system

If I'm reading this right, this seems like a really interesting and possibly lighter-weight alternative to Packer + Kubernetes. I like this and will give it a whirl.

No, you're not reading this correctly. This is not a Packer or Kubernetes alternative. It's merely a container runtime, potentially suitable for integration into a larger system like Kubernetes.

This is a different thing really. It's a container runtime used under the hood by Docker; rather than a container orchestrator.

I know this is for rkt but it gives a good comparison of container runtimes. https://coreos.com/rkt/docs/latest/rkt-vs-other-projects.htm...

If you read their roadmap, they have a reason for spliting up the engine into a few components, this being one of those.

Totally unrelated. Totally. Pure coincidence. Really. ;-)

https://github.com/kubernetes-incubator/cri-o

Reply on news.ycombinator.com