CoreOS Overview, Part One | Hacker News Reader

CoreOS Overview, Part One

deis.com

98 pointsnslater36 comments

Pardon my ignorance... Is CoreOS an alternative to Docker? And if yes, what are the tradeoffs between the two?

Its a company that develops a bunch of software and some of it makes containers easier to use. There's also their distro coreos. They are working on rkt which is like docker from what ive seen.

The feeling I get from them is their trying to have their own branded container stack.

IMO their container stack is much better decoupled than Docker's.

In Docker's early days they made a big deal about how a standard container spec could revolutionize software the way the shipping container spec revolutionized the transport of goods.

An independent spec never emerged though. The Docker monolith got bigger and bigger. They threw out standalone mode, so instead of being able to just be a container runtime, Docker basically owned the whole host, and your logging, networking, process monitoring, etc. all had to be done Docker's way.

CoreOS created the App Container spec (https://github.com/appc) to deliver on the promise that Docker started with. They eventually shamed Docker into becoming part of the Open Container Initiative (https://www.opencontainers.org/). Hopefully that means we'll get to a point where application images can truly be shipped anywhere and run anywhere, without having to give any one company control of your whole stack.

As far as I can see rkt has made no effort to implement the opencontainers spec, unlike docker. Meanwhile Docker is implementing all the pieces you are asking for, eg containerd and runc are standalone runtime parts which will break up the monolithic structure, using the spec.

I spend significant time helping the effort (Brandon Philips, CoreOS).

But, today there just isn't anything to implement in rkt. The primary focus of OCI has been a rootfs on-disk and a json file that contains details of cgroups, namespaces, etc. This is fine work to be doing but it doesn't tackle the more difficult and useful work of creating inter-op for developers up the stack who don't care about the internals of containers.

Here is a high-level post on the state of the world: https://coreos.com/blog/making-sense-of-standards/

Here is a video that I gave that summarizes some of my thoughts: http://containersummit.io/events/sf-2015/videos/container-ec...

A more detailed discussion on the OCI mailing list from earlier this year: https://groups.google.com/a/opencontainers.org/d/msg/dev/uo1...

tl;dr Container formats are important to get right. But, today the OCI effort is largely irrelevant for people packaging and shipping their code in a application containers.

edit:

Also when OCI finally makes it to a stable release we will likely add tooling to help import into rkt: https://github.com/coreos/rkt/pull/1509

CoreOS is a Linux distro based on Gentoo. It is noteworthy for being very small, built for "all-or-nothing" automated updates, and geared for clustering.

Basically if installed on disk (you can also boot it over the network or from a USB stick etc.) it updates by having two system partitions - one "known good" because you've successfully booted from it, and one used to download new updates to, and then will (by default) automatically reboot and switch to update.

The system you end up in is basically systemd + Etcd + Fleet (with Kubernetes on the way) + Docker and rkt, with very little else included.

If you want to spin up a large cluster where everything will run in Docker or rkt containers, that's something CoreOS is well suited for.

CoreOS is a fork of ChromeOS, not Gentoo. Both just happen to use emerge. It might poach some Gentoo packages to save work (I don't recall off hand) but build and usage is very different from Gentoo. Importantly, emerge is build-time trivia and has no bearing on the final product; there is not even an emerge binary in a CoreOS image.

(I've seen this meme before, with people - not you - dismissing CoreOS because it's "based on Gentoo," due to the presence of emerge in the build. Pass this clarifying point on to your friends.)

CoreOS is built using Gentoo. Gentoo is a meta-distribution whose main contribution is the portage tree which is a collective of ebuilds that describe how to build each individual package, the dependencies, the configuration flags etc. Literally every user who uses Gentoo builds and produces their own linux distribution, with their own CFLAGS, USE flags etc. ChromeOS is built using Gentoo, so is CoreOS. Neither ChromeOS nor CoreOS are Gentoo based distros because Gentoo is a toolkit to build distributions, not a distribution itself.

But it seems that CoreOS is more suitable as being the underlying OS on which containers are run, rather than being the basis of a container image?

Container images do not need operating systems, and I will forever be angry with Docker for tainting the concept in popular usage. Research a static binary as a container to see what I mean.

Nobody who built containers before Docker came along shipped an operating system in the image. Now we have gigabytes of Ubuntu alongside nginx thanks to Docker. Cool.

This. With other sandbox container solutions, you just start it. Docker's is mixing a sandbox, a deployment solution and more together. This might be useful for large deployments, but for smaller one it's too complex.

for sandboxes you don't need containers and for resource management, neither. The only thing which docker gives you, is that it enforces immutable disks, which cgroups can't do, can't yet do or whatever. however when even with docker you need to ensure that your program won't mutate the filesystem so anyways nothing special.

Enforcing an immutable filesystem is actually pretty easy to do, just don't give write permissions to the user under which the program is running :)

What Docker integrates is a layered filesystem, in which the application can mutate the filesystem, but only it will see those changes (and they'll go away when the container is destroyed).

oh yeah sorry. however on some linux filesystems you could also use `chattr +x` however most things won't play well with that ;) especially not programs which create logs / pids / whatever.

You can mount the base filesystem read only in the container as of 1.10, with tmpfs mounts for where you need ephemeral mutation.

Disclaimer: I work at Docker.

> Nobody who built containers before Docker came along shipped an operating system in the image. Now we have gigabytes of Ubuntu alongside nginx thanks to Docker. Cool.

First: I believe shipping the entire distro in each container was in fact the most common use of containers before Docker, since openvz/vserver/lxc were focused on "mini-servers" (to this day you can still rent openvz-based vps). So I think your statement is incorrect.

I also think your (eternal?) anger might be misplaced. Yes, today most people who use Docker build containers with unnecessary distro bits in them. But before Docker, 99% of them didn't build containers at all, and struggled with non-repeatable deployments and dependency hell. Even better, now all these people use a standardized Dockerfile which can be changed and improved over time - for example to move away from a heavyweight distro like fedora or ubuntu, to something like alpine or even more minimalistic.

Incidentay, we have hired Natanael Copa, the awesome creator of Alpine Linux and are in the process of switching the Docker official image library from ubuntu to Alpine. You can help us with pull requests to https://github.com/docker-library if you want :)

Even one such pull request could go a long way to promoting smaller containers, since those images have been downloaded over half a billion times.

> Yes, today most people who use Docker build containers with unnecessary distro bits in them.

>... to move away from a heavyweight distro like fedora or ubuntu, to something like alpine or even more minimalistic.

The distro package management is one of the biggest reasons why people still stick with heavyweight distros like Ubuntu.

This is why DockerSlim [1] was created (at one of the Docker hack days, by the way :-)) It lets you use heavyweight distros as-is eliminating everything you don't need. You get the best of both worlds where you don't have to change your process and you still get small images (7, 10, 20 and even 30 times smaller than the original).

[1] - http://dockersl.im

P.S.

It is safe for use in production :)

There's a packer manager in alpine. Maybe not as complete as ubuntu but pretty decent imho.

Yes, it does have a package manager, but... , like you said, it's not as complete and it is different, which means you need to put in extra efforts to convert and make things work (also don't forget about a potential lib C compatibility fun). Why go through all this trouble though? Why not use the distro you are already using as-is and still get Alpine-like image sizes? One nice thing about Alpine is its hardened kernel, but you won't be using it in your Docker container. Alpine is nice, but it's not worth it when there are other alternatives that don't require extra time and effort I'd rather use to build a better product.

And if you are using 3rd party Docker images Alpine won't help :)

Has something like nix been considered for alpine? I think it would be really great for use in containers since it has the built in ability to garbage collect unused dependencies.

That's great work, you may also look at https://github.com/rdicosmo/ami-thinner which is described in http://www.dicosmo.org/MyOpinions/index.php?post/2013/02/08/...

That is awesome! Congrats ncopa. Incidentally I ported go and then docker to Alpine Linux (musl and hardened toolchain fixes) almost 2 years ago:

https://github.com/alpinelinux/aports/commit/abfdfd32d2c6818...

https://github.com/alpinelinux/aports/commit/4248d1a88bcde1d...

I said "built containers." Not rented a VM which happens to be run atop container-style virt. Since Docker is positioned toward the former, not the latter, but due to your company origins and history ends up looking like the latter, thereby training a generation of developers that /etc/hosts and shells belong in production artifact containers, my anger is correctly placed.

Docker has regressed the container term completely and irreversibly. I'm glad you're switching to Alpine (and kudos on spinning a negative comment like a top), but you skipped over that I said static builds, as in: the container doesn't carry hundreds of files to support a full Linux distribution with the weight of legacy support atop it. A container should be one binary. Dependencies are one thing, and yes, they often have to ship in the image. I don't find /sbin/init or /etc/nsswitch.conf when I open up JARs built by the average CI, though, so what the hell are they doing in Docker containers? Now the term is so blown out that my comment alone is probably causing crossed eyes and tilted heads among readers, because nobody really knows what a container is any more.

You know this opinion exists, too, because certain folks have reminded you numerous times in the political theater that containerization has become behind the scenes. Also largely thanks to Docker. (Not solely. Largely.)

`FROM scratch` should have been the default. Come on. You know it just as much as I do, Solomon. Imagine if you had shipped a build chain to automate the same dependency work that Google did to build containers, instead of just dumping ubuntu-minimal into a container so dynamic linking and such works and finally trapping us in a hell where Alpine is seen as better.

Almost no containers have /sbin/init in. An alpine container has little in it. But C applications do generally require some files, eg config. Docker does encourage people to put as little as possible in, but alas there are still people who shell out as they do not have libraries do do what they need yet... including Java programmers.

Disclaimer: I work at Docker on unikernels. You might like this approach to containers.

You don't even need unikernels, and as much as I loathe myself for saying it, I find myself agreeing with a few of Cantrill's points regarding unikernels in prod. Not all of them, and I think it's worth exploring, but there's a spectrum here: on one end, unikernel app containers, and on the other full jails. The Google approach with minimal containers that still act Unixy and Posixy but carry very little distribution overhead is somewhere around 0.1 on the spectrum.

But we call [0.0, 1.0] the same thing now, so the tooling has to support all cases, and now just pulling a Docker image down might cost me gigabytes of network transfer that I don't need to spend. Combined with the HTTP-strong architecture, Docker encourages fat containers but then falls apart on them at higher sizes because of the HTTP mechanics.

Example: I need to ship about 1 TB of static data alongside my app in a separate container. Try that in Docker. The answer will invariably be Docker not being designed for that, but I thought Docker encouraged the jail-style fat container case? What happens when the jail you're renting out because that's a container too actually uses a bunch of space and you try to `docker commit` into whatever you're managing your containers?

You shouldn't be using containers for that, but features of Docker encourage it, and thinking fat like that has numerous pitfalls. (My example is borne from reality: I watched Docker wash out from a team in two days once they tried to commit a 100+ GB container. I assume the docs guided them there.)

If you would like an official Zulu JDK container I am willing to contribute time, from my Alpine-based Smallest JVM containers in the world™ http://j.mp/tinyjvm

Yes please! That sounds fantastic, thanks for offering.

I take a small issue with "now all these people use a standardized Dockerfile"

And many folks don't, because the amount of unreadability from having to concat 20 statements in a && fashion to cut down layer size gets out of hand. This leads to folks then having to use scripts to clean the layers, which leads to reuse in things like Packer, which leads to even more problems when the base image changes culturally.

I'd also argue the that it's a matter of familiarity. Every time we change the base image culturally, we have to retool for a different method of making the things in it (apt/dnf/yum -> apk), so while large distro images are bad, they're familiar and predictable (with a mature package set and releases) to what we're used to doing on servers.

Or folks use another tool to abstract that (like Ansible or Chef), which brings us right back to not using Dockerfile.

I would imagine the most important thing from a distro is the support from the downstream vendor (i.e: canonical or red hat) that comes with it. Downloading and compiling from source to a minimal distro (like what alpine does it seems) is nice and all but would be limited for toys/dev deployments and not sure how i would get my CTO convinced to use those images instead of officially supported one from a reputed vendor cause the benefits are a few mbs (and freeness obv) gains

> Nobody who built containers before Docker came along shipped an operating system in the image. Now we have gigabytes of Ubuntu alongside nginx thanks to Docker. Cool.

This couldnt be more false - other systems had containers capabilities for years now (FreeBSD Jails, Solaris Zones, IBM LPAR's, Linux OpenVZ and so on) and since forever it was done in a way of running another full OS inside a container. Stripping that down to single processes was possible and done by some people but rarely, because it was relatively hard (since it often required hand-crafting the 'packages', hunting down single libs and their requirements using ldd and the like). I myself was running both full OS jail containers on FreeBSD and stripped down single process/service jails more than 10 years ago.

As I told Solomon, this confusion is due to muddying of the term. I'm connected to the Google school of thought on this (where basically all the billions of containers in prod are `FROM scratch`, because they did the work you call hard to ship a process as a container), whereas jails are the other school of thought. I would contend that they are different enough to deserve different terms, and this thread of correcting what I meant, and explaining jails to me even though I would think it pretty clear I understand them, is evidence enough. We called those jails when FreeBSD was carrying that torch. I never once heard them called containers until now. That is my point.

Docker is not for the jail case. If it were, it would look like OpenVZ and come with completely different architecture and management tooling. It's for deploying artifacts to prod, and my entire point is that starting from `FROM ubuntu` for even a Go binary has muddled this picture irreversibly. Now I have to have this argument, which makes no sense, because we now take it as fact that containers and jails are the same thing, and anyone who says otherwise deserves to have very basic things explained to them carefully and rather condescendingly.

This reply applies to everyone who is calling me wrong. I spent last summer employed by CoreOS, have worked on Google Borg, and spent the first half of my career on the production artifact toolchain. I'm not just farting into the breeze here with my two personal servers and opinions.

Not true. Managing OS images is the killer app for containers.

Containers power the Heroku platform and it supports millions of apps by containerizing OS images.

When both the bamboo stack and cedar stack were available, we would:

Run a base AMI and update it frequently for kernel, security and performance updates.

Download a bamboo (Debian) OS layer

Dowbload a cedar (Ubuntu 12.04) OS layer.

Layer app slugs on top of which ever stack the customer wanted.

This was amazing for long term support of apps and backwards compatibility.

There are infinitely more rails apps out there that deeply depend on the OS layer than there are static binary apps.

Docker deserves and gets all its credit for making this technique available and easy for everyone.

To answer this more explicitly than the other comment: Yes, CoreOS is intended as the host system for containers.

Inside the Docker (or rkt) containers you can run whichever distribution or static binary you like.

The combination of CoreOS+Kubernetes+Docker+Alpine Linux (inside the containers) works pretty well in my experience.

You had me interested until you said systemd.

No. CoreOS is a Linux distro focused on running applications packaged as containers (Docker or anything else) on top of it. It is so focused on running application in containers that it does not even have a package manager to install packages (Hence, they have different update/patching mechanism). Their whole philosophy is you run CoreOS as the base OS for your cluster, and run your applications on top of it packaged as containers. It also provides a bunch of tools for your cluster management and service discovery across containers/applications in your cluster.

PS: the above is a very summarized, focused answer to your question. CoreOS is also much more than what I have mentioned.

In addition, CoreOS has its own version of a docker-like thing called rocket https://coreos.com/blog/rocket/

I personally found this blog post a bit of a a-ha moment in terms of understanding what docker is vs. CoreOS (and, at the time I was also researching what kubernetes is and how to run it)

Reply on news.ycombinator.com