Fedora CoreOS, Red Hat CoreOS, and the future of Container Linux | Hacker News Reader

Fedora CoreOS, Red Hat CoreOS, and the future of Container Linux

coreos.com

145 points19h64 comments

This is exactly what I thought would happen when I switched off of CoreOS earlier this year. I recently switched back[0] since according to their blog post[1] Container Linux would live and not be replaced by their Atomic project... I thought I was a fool for overreacting.

I don't want OpenShift, it looks like a bloated clusterfuck. I don't want Atomic either -- Container Linux had all the pieces I needed and not much else, along with innovation just where it was needed (the update engine).

It might not mean much to Red Hat in the economical sense right now but Red Hat has gotten a red strike in my book from this. I won't forget. Corporate double speak/renegging on acqui-hired goodwill normally doesn't get me so riled up but man if my jimmies aren't rustled right now.

What does everyone else use for their server distros? Is debian the way to go? Ubuntu seems bloated but maybe I just don't know enough about the fat that is cut out of Ubuntu Server? CentOS and Fedora are stable but they seem like they update too slowly, is my assumption wrong?

[EDIT] - This post is likely an overreaction (again?) -- as pointed out by others the linked thread is from May 2018 -- so my reaction is very much delayed. If I had seen this thread when I made my decision to switch back to Container Linux, I wouldn't have.

I'll likely be moving to flatcar linux[2].

[0]: https://vadosware.io/post/yet-another-cluster-reinstall-back...

[1]: https://www.redhat.com/en/about/press-releases/red-hat-acqui...

[2]: https://www.flatcar-linux.org/

I think many are using Alpine linux [1]. It's very light weight and seem to be well suited for the task.

[1] https://www.alpinelinux.org/

does Alpine Linux provide Transactional Updates?

Try NixOS for something that has something close to transactional updates.

Wow it looks pretty neat. NixOps looks also really cool, you can deploy in VirtualBox and to the cloud. Reminds me of the old Hashicorp Otto.

Using dockerTools in Nix means you get distroless images!

The 'virtual' version is a mere 32MB. Neat.

Alpine is well known as the OS for containers[0]. BTW the guy pictured is Natanael Copa, the creator of Alpine Linux.

It's a fantastic distribution, and it's a pleasure to use.

[0]: https://thenewstack.io/alpine-linux-heart-docker/

I don't read that as openshift being the replacement. I read that as openshift being the first target of integration efforts, with the replacement coming afterwards.

OpenShift is really garbage. Everything red hat adds just makes it worse, and it's a lot slower. The installer is absurdly slow also.

I am worried that I'm shitting on it without using it, but every time I even take a look at the documentation, I just can't stomach it. There's so much happening, all over the place, and it all looks like it was created to be composable, but it seems very non-optional (as in you need to learn 4 OpenShift things before you can do the thing you wanted to do).

Maybe there's someone out there that loves working with it and feels like it's worth the effort but I haven't seen many posts from them. Makes me feel like they're all stuck in corporate dungeons toiling away using stuff they were forced to use.

IMO OpenShift is RedHat's attempt to embrace extend and extinguish vanilla Kubernetes because it threatens their Enterprise OS domination. The host OS for Kubernetes clusters is mostly irrelevant.

It's like Kubernetes... forked from it, but adds all this other shit while they continue to just say it is Kubernetes under the hood. Technically true, but once you go to the OpenShift you're pretty much locked into RedHat's Kubernetesesque-world.

OpenShift predates Kubernetes by a bit, and IMO they operate at different levels (OpenShift is "deeper" and overlaps in some spots with Kubernetes). Also, I'm just about 100% sure no one is stupid/brave enough to challenge Kubernetes' current dominance in the container orchestration space right now. Kubernetes is complete enough and not-bad enough to be the defacto choice right now, and I doubt much will change -- plus Google is backing it, along with the CNCF, there are so many companies with a (in)vested interest.

I do think Red Hat replicates features that Kubernetes does well and trying to do those things well but they operate at different levels fundamentally -- Openshift is like a bunch of individual components that work together (usually at a lower level than Kubernetes does) and Kubernetes is like one coherent platform that smoothes over all the lower-level stuff (CRI, CSI, C*I)...

OpenShift is Kubernetes. A few years ago, they realized that Kubernetes is the future and rebuilt their product on top of it.

OpenShift was re-written to be based on Kubernetes as of version 3 in 2014: https://blog.openshift.com/openshift-v3-platform-combines-do...

I am currently moving all of my infrastructure to OpenShift and I love it - and I chose it after carefully evaluating the alternatives.

It's Kubernetes plus a PaaS platform that takes care of the annoying parts - deploying a cluster (using Ansible), container builds, triggers, deployments, a nice UI... Couldn't be happier.

Red Hat is a major Kubernetes contributor and OpenShift is barely lagging behind upstream k8s. It feels very polished and the documentation - while a bit overwhelming at times - is extremely helpful and extensive. Instead of forking Kubernetes, they only ever add new functionality while simultaneously upstreaming it. For example, the Kubernetes RBAC mechanism was contributed by Red Hat.

I use CentOS, it updates about a month after redhat doses for the OS and a bit more frequently for most packages in my experience. It's also what >80% of my company uses for there server infrastructure.

I happily use Ubuntu on a few thousand servers running 10s of thousands of containers. We're a polyglot shop with lots of java and lots of go/node and my comfort level with ubuntu is high.

I am also a big fan of Ubuntu Server and Mark Shuttleworth. The guy does not get the proper credit. He's basically running a charity for hackers and startups yet, for some reason, is not very popular on HN.

He's a risk taker with Unity/Ubuntu Phone/other things that didn't pan out, and people treat it like Google abandoning their products. Canonical has money, but not Google money.

People also get overly bent out of shape for having Amazon search integration with the desktop at one point, which I did not like, but it had a clear, functional way to disable that function.

At least, those are my theories.

Their revenue of ~$126M in 2017 is very modest compared with its rivals Red Hat (~$3B) or Microsoft (~$90B). My biggest fear is that one of them may take over Canonical and shut down Ubuntu.

Thanks so much for sharing -- I think I'm going to give Ubuntu Server a try -- Canonical has done so much for open source (as noted by the other comment) and a bunch of innovation lately -- LXD is something I've actually wanted to play with as well as a virtualization option, which I actually failed to set up on Container Linux[0]. I swear I'm not trying to blog-spam, I just legitimately have been dealing with this stuff lately and have been effectively swimming upstream by not using Ubuntu -- looks like I need to stop.

I was mainly worried I'd be spending time downloading noveau/radeon drivers and associated packages on a server with no attached GPUs. I've been leaning towards languages that compile fat binaries (and running with docker regardless), so this is why I'm a little wary of Ubuntu bringing too much to the table. Also, it's been a long time since I ran Ubuntu on a personal machine, I am still a little worried about the risky the dist-upgrade process can be.

Basically all I feel I need is ufw, docker, ssh and was worried that Ubuntu brings too much along for the ride.

[0]: https://vadosware.io/post/trying-and-failing-to-get-lxd-runn...

Server is based on a very minimal seed. There are no graphics drivers or anything X or desktop related. The seed is ultimately very close to what a cloud instance of Ubuntu will include if you want to give it a spin.

There's also openSUSE Kubic. https://kubic.opensuse.org

That is a cool project:

* Transactional updates. No interference with the deployments. You can rollback to any previous state.

* Smart separation of /, /etc and /var, using volumes and overlays properly.

* Using RPMs! I can tailor my installation with traditional RPMs, and the result will be updated at once.

* Zero maintenance. This is kind of magic for me, you create the initial deploy and the system upgrade itself, and rollback if a problem is detected. I wonder how well this works IRL

On the downside is still a bit new, but I found more information here [1]

[1] https://kubic.opensuse.org/blog/2018-07-09-kubicatosc18/

Chris from Kinvolk here.

Flatcar Linux is generally available and we are committed to keeping it as a drop-in replacement for Container Linux for the long-term. We were excited about the idea of CoreOS when it was announced in 2013 and think it's a project worthy of sustaining.

Happy to answer questions about it.

Thanks for mentioning this, it looks cool.

Link, for those curious: https://www.flatcar-linux.org/

I'm not particularly fond of how OpenShift is put together, but it does seem like you're overreacting somewhat.

CoreOS is pretty well aligned with Fedora, so it makes sense for it to go there organizationally. And while it's inevitable that everything in Red Hat's orbit will have a clear path to OpenShift (and probably Ansible), I'm hopeful that this won't add any noticable overhead to CoreOS/CL itself.

I took a look at the coreos fedora project[0] as listed by you and others, but it looks immensely experimental right now. From the FAQ[1]:

> Fedora CoreOS is under active development and there are currently no downloads available. However, you will be able to download the images from one canonical location via the CoreOS and Fedora websites. The discussions around this development will happen on our community channels.

I actually really like Ansible -- it plays well with other tools, scales from small to large infrastructure management needs, and has great documentation with enough escape hatches to make hacky (but functional) solutions possible. I have less of a problem with ansible and more being forced to move more and more into tools that push me towards OpenShift when it doesn't make sense for me.

[0]: https://coreos.fedoraproject.org

[1]: https://coreos.fedoraproject.org/faq

Yea, I'll 2nd that. Ansible is immensely pragmatic and capable. Been very happy with it.

It's slow and hard to debug :( (at least in my experience - setting up OpenStack with it)

I used to prefer Chef, but then with CoreOS the dream started to come true, just pack up everything in a container, and 12-factor configure it and that's it, and now I just can't wait for k8s to eat the world, and we'll be back full circle with helm instead of yum/apt.

Not ad-hominem but what do you know how to do? Package other peoples stuff in a config + install and then run scans against it?

I don't get what you mean, could you please elaborate a bit?

Red Hat uses Ansible to set up the Kubernetes/OpenShift cluster itself.

Fedora: Only if you really like breakage and have staff to deal with integration.

Happily running OpenShift Origin on top of CentOS.

> What does everyone else use for their server distros? Is debian the way to go? Ubuntu seems bloated but maybe I just don't know enough about the fat that is cut out of Ubuntu Server? CentOS and Fedora are stable but they seem like they update too slowly, is my assumption wrong?

CentOS/RHEL stability is exactly the reason why I use it. Red Hat spends a lot of money on quality assurance, and it shows. It's a very different experience than Ubuntu, which has a lot more breakage.

The Kubernetes/OpenShift packaging is mostly independent of the host operating system. My applications inside the containers mostly run on Fedora, CentOS + Software Collections and even bleeding-edge Ubuntu, but I'm happy to run a stable OS underneath.

CentOS/RHEL 7 moves a lot faster than previous releases, too. They even rebased OpenSSL for 7.4 to get HTTP/2 support, all while maintaining ABI compatibility.

Fedora is - by design - a lot less stable than CentOS/RHEL - it's the fast-moving upstream project.

If you're on coreos, probably just wait till early next year to see what the new OS will be like, then make a decision.

If you have a deployed fleet of CoreOS instances, FlatCar[1] looks like a viable alternative. This article from NewStack gets into the details.

[1] https://www.flatcar-linux.org/ [2] https://thenewstack.io/flatcar-linux-the-coreos-operating-sy...

Last time I saw flatcar they were in some sort of invite-only stage -- super glad that they've released and I'm excited to switch.

Chris from Kinvolk here.

It's open for general use now. :)

Maintaining an operating system is a massive undertaking. Can Kinvolk guarantee that they will support the distro after Red Hat stops working on it?

That's a very valid question and one we also asked ourselves before taking on the task. There are three main considerations that made us feel this is doable. Firstly, Flatcar is a minimal Linux distro, with more effort made to reduce the number of packages rather than expand. Secondly, Flatcar has not only Container Linux as an upstream, but also Gentoo, which is also a common upstream for Chrome OS and Container-optimized Linux, so well tended to. Thirdly, the technologies (kernel, systemd, containers, low-level Linux userspace) are our core focus as a company. These three considerations are what led us to feel confident in picking up the project and have the motivation to do so.

The way I personally see it, if you were ok with using an OS from a team of veteran Linux/open-source contributors in 2013 you should be ok with using one from a team of veteran Linux/open-source contributors in 2018. But this time it's from a team that does not take venture funding.

Lastly, it is telling that the ppl that have the most confidence in us pulling this off are the ppl we've worked with as clients or through open source collaborations. We're happy to start there and prove ourselves along the way to other.

Thank you for your response!

I wish you the best of luck, choice is important for the ecosystem.

Why would one want to do that?

As to why we'd do it in the first place, the answer is that our mission as a company is to support foundational Linux technologies for cloud-native infrastructure. We're not looking to create new technologies, but to improve and support those that exist and are widely used already. Container Linux fits that description to a T. Additionally, as we mention in the FAQ, we were already being asked to support Container Linux. Thus, we knew that there was demand and it fits our company mission.

So when the acquisition was announced, it was a rather easy decision to make; one that we'd already been considering. But knowing that it was also likely to face changes under new "owners" allows us to also be in the position of preserving a technology that we feel is fundamentally sound.

In general, I'm pretty happy about this. The more I used Atomic Host, the more I liked it. While it was a pretty dramatic departure from Fedora's dnf/rpm world, I really like how rpm-ostree handled the Atomic update/rollback stuff. I thought it was a nice middle ground between a traditional Linux distro and Container Linux's "just slap a new image down to disk for upgrades". I think there's a time and place for both and Atomic Host had a nice merging. Container Linux with the rpm-ostree tooling should be really nice for orgs that want to customize their distributions.

I'm only a novice Ignition user though, so I'd appreciate it if anyone could tell me what benefit it has over kickstart for provisioning nodes and clusters that makes Red Hat want to keep using it. Kickstart clearly has way more configurability for deciding how you want the host to be provisioned (at the expense of added complexity), so the only place I see Ignition adding value is that it's a really easy way to group nodes and assign roles by hostname, like a lightweight Cobbler install or something. Am I missing something else?

Canonical has timed its announcement of Minimal Ubuntu (29MB) really well. [0]

[0] https://blog.ubuntu.com/2018/07/09/minimal-ubuntu-released

I think by the time they figure out what Fedora CoreOS will be, they will have basically missed the boat for non-Red Hat customer adoption.

Basically it will only really matter for people running OpenShift.

I'm cautiously optimistic about Fedora CoreOS, but am worried that it brings with it bloat that wasn't present in Container Linux.

What is the story with rkt? Is it deprecated?

Yes, they announced at the Red Hat summit that they'll be phasing out development of rkt, though of course will do nothing to stop outside development. Between docker and cri-o, I think they decided that the need for a third supported and actively developed runtime wasn't enough to justify the effort. I'm excited to have the rkt developers work on cri-o, assuming that's where they're going.

Bummer, I preferred it in some ways.

Where does SilverBlue fit in? Is it going to be the workstation distro of Fedora CoreOS that bundles a GUI and some userland tools?

https://teamsilverblue.org/

I’ve been interested for about a year and gave Atomic a couple test drives, but with the CoreOS acquisition and so much overlap between these projects I’m wondering if RedHat actually has a plan for each of them.

> In-place upgrades from Container Linux to Fedora CoreOS will not be possible

...

Why not use the flatcar-linux model?!

We've crossed the point where running Kubernetes on CoreOS was easy.

Maintaining Kubernetes on Ubuntu with docker (vs rkt) is now just easier to handle.

You mightwant check Alpine or ArchLinux. They are fast, reliable and updated regularly.

In my experience Arch Linux isn't that reliable. I had a few packages which where broken for weeks and I couldn't manage to downgrade them due to the rolling-release nature.

With Fedora there are less feature updates and I can still go back to a previous release. And if I really need the cutting edge, I can cherry-pick packages from Rawhide.

I think the title needs to be fixed, it's a tad sensationalist. The Google Groups message is just "Container Linux project update" and is dated May 18th. Since then, there has been more clarification [0], [1], [2], that really calls out a rationalization of the RH roadmap and where CoreOS fits. It might be fair to say that RH doesnt plan to support "CoreOS Container Linux" in exactly the fashion prior to acquisition, but it would be disingenuous to imply it is being wholly discontinued.

[0] https://coreos.com/blog/coreos-tech-to-combine-with-red-hat-...

[1] https://coreos.com/blog/fedora-coreos-red-hat-coreos-and-fut...

[2] https://fedoramagazine.org/announcing-fedora-coreos/

Thanks! A moderator updated the link from https://groups.google.com/forum/#!msg/coreos-user/PFHJQMJSRb....

Hold on, lets just buy the competition so we can rack in more enterprise clients. :/

With my experiences with RedHat, The only thing I've gleamed from it is that they are a mess with packages, and are their own 'standard.'

RH became irrelevant when everyone was moving to cloud shenanigans and they didn't update their pricing accordingly.

when even fortune50 have to drop your product they used for decades because of costs, that's a very clear indication of the end.

Source?

I have heard(both directly from customer and from friends at other orgs) of 3 Global 500 companies in the past 6 months walking away from OpenShift due to licensing costs. I am sure this will play out more frequently over the next 12-18 months.

Take some time to review RH's annual statement from 2017. Think about where they were in 2013/14, where we are at today and what things might look like in 2023. I am interested to see how they navigate the market. There is a fleet of Titantic like vessels plowing through the enterprise ocean right now...it will be interesting to observe what plays out over the next 3-5 years.

Call redhat and ask how much it would cost to do a "lift and move" strategy (i.e. just pack your legacy software in a vm and run it of the 'cloud')

They will do some crazy calculation and say, "well, aws will place the VM into a zillion machines, so that times our license per machine, plus our license per users per machine, you own us infinite dollars" ...it is that silly. If you don't trust me on this call them.

Reply on news.ycombinator.com