Fedora CoreOS Out of Preview | Hacker News Reader

Fedora CoreOS Out of Preview

fedoramagazine.org

121 pointsecliptik71 comments

The positive headline hides bad news for those of us who run CoreOS. CoreOS will be out of support soon(?). Migration instructions have been announced, but not written yet...

There is a CoreOS fork at https://www.flatcar-linux.org/ promising seamless continuation. I have no experience with or further knowledge about it.

Are you thinking of Container Linux by CoreOS (The company purchased by Red Hat)? Fedora CoreOS is a continuation of Container Linux.

See https://en.wikipedia.org/wiki/Container_Linux#Derivatives

Yes, sorry, unprecise terminology. When I say CoreOS I mean CoreOS Container Linux.

Fedora CoreOS is marketed to be a continuation. But it's not a seamless one, it requires manual porting. Detailed instructions don't exist yet, but the list is not that short https://github.com/coreos/fedora-coreos-tracker/issues/159. While most people won't be affected by many items, a couple of days are easily burned for understanding, coding and testing. Coding probably the smallest part, but if everything goes without "surprises" in real life, I would be surprised.

This is generally one of the philosophical differences I see between the RHAT family of distributions and some others (those that I personally favor), that seamless upgrades are generally not well supported from each major version to next.

For example, and I'm sure I'm oversimplifying, and there are caveats; for years, I've maintained Debian and Ubuntu machines which I've used mostly staying inside of the bounds of the apt package manager, and since many years I've never had a need to question whether I could "dist-upgrade" from one release to the next. It just works.

It is a supported operation, and reliably so. At my current job, we use Amazon Linux, a distro which I understand to be mostly derived from CentOS with Amazon addons, (and I'm open to being corrected if anybody knows better, but...) one of the things we're coping with right now is that Amazon Linux 2 does not have an upgrade path from Amazon Linux 1. We are on the hook to rebuild those machines, it's a fact that has given us pause to seriously re-evaluate whether we want to stay on the same train, or get off and switch to another paradigm broadly.

I don't strongly see this as a decision that Amazon made, because I'm fairly sure it's also true that upgrades from eg CentOS 6 to 7 are unsupported. And CentOS apparently got this idea from RedHat Enterprise Linux upstream, which provides this helpful page locked behind paywall[1], but from above the fold you can tell at least one thing without logging in: there are only a limited set of cases where upgrades are supported, even with a paid subscription to RHEL.

[1]: https://access.redhat.com/solutions/637583

I've upgraded multiple Fedora machines from Fedora 19 up to the current Fedora 30 without too much issue.

Contrariwise, I've found various issues with Debian and Ubuntu.

That said, I rarely do dist upgrades of prod systems at all; I install the base os from scratch (pxeboot etc) and use automation to install the rest.

I’m the same. I never upgrade cattle, they get replaced and only pets are upgraded.

I will concede, Fedora has as good a reputation and it's what I run on my desktop at home! Good point.

Honestly I've had more success migrating between versions of the Fedora family than the Ubuntu family. I don't have enough experience with Debian to judge.

In an case I disagree totally with the assertion that Fedora isn't good at upgrades between versions.

CentOS 7 upgrade would be tricky to do automatically because of the systemd transition and the fact that any Enterprise user would likely have a ton of services they need to migrate safely.

> In an case I disagree totally with the assertion that Fedora isn't good at upgrades between versions.

Same here. I think that changed somewhere around Fedora 15, though. Prior to that it was a minefield. That's probably why it has such a bad reputation among those who've not tried recently.

My current desktop workstation started out at Fedora 14 and now sits at 31. It's gone through multiple motherboard and component upgrades over that time, too.

I said RHAT family, but honestly I did not mean to include Fedora. I've been upgrading my Fedora Core desktop since probably 26, it's not the machine that gets the most use, but it works at least as well through upgrades as the Ubuntu and Debian systems that I maintain.

But none of my servers run Fedora. I still think that the core use case for Linux is servers. (And that's not to say my servers couldn't run Fedora... but it so happens the OS distribution that we use at my $dayjob is usually Centos or Amazon, sometimes RHEL, occasionally Ubuntu, but for some reason absolutely never Fedora.)

I'm not sure that migrating to systemd is as hard as you think, I've done it on my Debian release train, and I've also done it with Ubuntu, after migrating from Sys-V to Upstart, no less even...

It's actually standard policy from what I understand on the distros that are in the RHEL family, that upgrades between major version of releases are officially not supported.

So, Fedora is nice, but if you're using RHEL, be prepared to rebuild everything from scratch every 2-3 years, I guess?

The same is true for Amazon Linux -> Amazon Linux 2.

They are, but there are some requirements to be fulfilled.

For 7.6 -> 8, see https://access.redhat.com/documentation/en-us/red_hat_enterp....

Thank you! I assume there are requirements to fulfill for the other distros which support it as well but they are probably not as well documented.

One thing which RedHat has always impressed me with is the quality of their documentation.

I've been upgrading the same Opensuse installation since 11.1, and I've moved it to 2 new machines without re-installing...

If we're bragging...

> upgrades from eg CentOS 6 to 7 are unsupported.

Upgrades from EL7 to EL8 are supported, though.

For Container Linux/Fedora CoreOS, in-place updates don't make much sense. Those are intended as immutable OS images. They get their configuration from a remote source specified on the kernel command line.

The migration work is dealing with changes in the configuration language.

Upgrades between CentOS are not officially supported. Upgrades from RHEL 6 -> 7 and 7 -> 8 are supported, but only for an extremely small subset of packages and systems setups. It’s not really worth it. PXE boot with custom kickstart and post config (e.g. Ansible) runs are a much better solution than the in place upgrades.

> The migration work is dealing with changes in the configuration language.

Yes, that's how all installations work for CoreOS Container Linux. The problem is the day when the autoupdate of the current distro stops (no more new images, and that's supposed to happen soonish, although no date has been communicated) your old configuration language will stop to work (unless you are happy enough not to used anything on the list of incompatibilities, which is pretty unlikely for non-trivial installations). And then you need to port and test.

In my experience coding the configuration takes at least 3 times as long as installing in a traditional way. That is a worthwile investment if you want to be sure what you have installed and want to be able to start identical machines later. But if the investment is broken by commercial/political decisions that's frustrating.

Well, I am not a paying customer, so I cannot really complain. I just have to decide whether I am happy with the flatcar fork or whether I port my configuration to Fedora CoreOS. At the moment they have no complete documentation AFAIK, So it's difficult to make a decision.

We started Flatcar Container Linux to provide the option of continuing as is. We don't like to see perfectly good software be discarded simply because of an acquisition. We understand the hassle with porting configurations and have found a good number of organizations who are supporting our effort through support contracts to continue in the manner they intended when they chose to use CoreOS Container Linux.

We also released the update service as an open source project, Nebraska (https://github.com/kinvolk/nebraska). That was something that was closed source with CoreOS. This allows you to be in control of your updates.

>We are on the hook to rebuild those machines, it's a fact that has given us pause to seriously re-evaluate whether we want to stay on the same train, or get off and switch to another paradigm broadly.

That's part of the selling point of Kubernetes. Your base OS needs to have just enough to run Kubernetes so upgrading it (ie: nuke and rebuild) is easy. You containers on the other hand can be upgraded piecemeal and are built with code. It's more things to handle but makes each thing smaller in scope.

From what I've seen so far, it is a continuation in name only. Many of the things that made CoreOS nice have disappeared like locksmith, and other stuff has been made incompatible for no good reason, for example dropping docker...

Fedora has preferred podman for a while, it's not unique to Fedora CoreOS. Much of that has to do with Docker's choices.

moby-engine https://koji.fedoraproject.org/koji/packageinfo?packageID=27...

docker https://koji.fedoraproject.org/koji/packageinfo?packageID=51...

podman https://koji.fedoraproject.org/koji/packageinfo?packageID=26...

> have disappeared like locksmith

Locksmith[0] implementation is tightly coupled to the specific update daemon, so it can't be directly re-used outside of Container Linux or without update-engine[1].

Its logic has been ported over to Zincati[2], which performs reboot management on top of rpm-ostree[3].

[0] https://github.com/coreos/locksmith

[1] https://github.com/coreos/update_engine

[2] https://github.com/coreos/zincati

[3] https://github.com/coreos/rpm-ostree

Last I heard Docker still doesn't support cgroups-v2, 4 years after it was merged into Linux.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

> but if everything goes without "surprises" in real life, I would be surprised.

Do you also shave all those, and those only, who do not shave themselves?

Is there a firm date for this? The original statement was "at least to the end of 2019". My employer uses CoreOS Container Linux extensively. It's really frustrating not knowing when the hammer will drop.

I'm not sure Flatcar Linux will make it through our security team. I'm "looking forward" to the migration guide to see how much pain i'm in for!

And a firecracker is guaranteed to not explode for at least 3 secs after its lit.

Chris from Kinvolk here.

We're happy to talk about what you need on the security front. Some of the folks working on Flatcar Container Linux have a very strong security background; worked on AWS' EC2 security team, do regular pentesting for distributed systems[1], have reported dozens of security issues to upstream projects packaged in Flatcar Container Linux, including the kernel.

We've just now started breaking away from the upstream project, and updating packages. Addressing any open security issues is front and center in our efforts. If you have concerns we'd love to hear them.

We worked with CoreOS team for years (was our founding project) and they trusted us on the security front. We feel that if you trusted CoreOS and know our team + background, you should have just as much trust in Kinvolk.

[1] https://www.youtube.com/watch?v=ze1vgh8sjlE

Hi Chris,

Thanks for the informed response. To be honest I hope Flatcar does well, I notice the Docker version has been updated on your edge release - long overdue from the upstream project!

I will let you know if our security team has any issues if and when they look at it.

Thanks for giving everyone the option of staying on Container Linux!

The whole point of CoreOS Container Linux was to deliver a steady stream of security/software updates. We've been eager to update packages for Flatcar Container Linux but have wanted to maintain as much compatibility as possible for as long as possible. Fairly soon, however, we'll be introducing an updated kernel and user space (systemd, Docker, etc.) into the alpha channel. For us, this will mark the point where we feel like we're fully taking the reins from CoreOS and carrying forward the original objectives.

Happy to answer any questions about Flatcar Container Linux (https://www.flatcar-linux.org/).

It should be noted that we went a step further than CoreOS by releasing the update service we use to manage updates as an open source project, Nebraska (https://github.com/kinvolk/nebraska).

Ignition is a killer feature, so much better than cloud-init, glad to see it surving and being actively worked on.

Is this the best solution if I want a standalone bare metal server that can spin up VMs? I'm looking for the closest analog to for example, an open source digitalocean style platform. Also interested in something that lets me manage CUDA/TF/PyTorch jobs. I know and understand docker and kvm but not kubernetes yet. TBH I find this entire space totally confusing.

If you want a standalone bare metal server that can spin up full-fat VMs that will live for a long time with some expectation of persistence, you are probably better looking at something like VMware ESXi (free for individual use on a single host), Joyent SmartOS or something based on KVM, Xen etc.

If you only want to run lightweight containers for comparably shorter periods of time and worry about data persistence through some other means, then things like Fedora CoreOS, VMware Photon are designed for that purpose.

> Is this the best solution if I want a standalone bare metal server that can spin up VMs?

No; Fedora CoreOS focuses on containers, not VMs. What you want is OpenStack, it seems.

OpenStack? For a standalone bare metal server?

No. oVirt makes much more sense for that case: https://ovirt.org/

OpenStack is a lot more complexity for a single box...

To be completely honest, for a single server, why not just use libvirt's commandline utilities?

That's fair, if it only is "a" server, then one has to weigh pros and cons. I said OpenStack because of the speed of spawning VMs ready to go, cattle style. Traditional VM platforms haven't really been able to do that.

Take a look at proxmox.

If Ubuntu is acceptable, Canonical LXD/LXC is worth looking at - containers, but full OS ones. Just so simple to configure and use, and minimal overhead.

Have used it without much problems for all kinds of workloads.

LXD will fully support KVM soon, too.

You have two options:

1 - https://dcos.io/ 2 - https://rancher.com/

You have a lot more then just two

CentOS 8 (or Fedora 32+) with Cockpit. Gives you a really nice UI for deploying VMs and Containers with a few clicks.

Could also look at a direct hypervisor, such as OVirt (upstream of RHV).

oVirt is a management platform on top of CentOS 7/8 and KVM. You can (and I have done) use Cockpit and oVirt on the same box.

I understand that. I should have explained a bit more than what I said. I meant instead of having a standard system with KVM, like using a normal CentOS host and running oVirt engine on top of it, use the oVirt Node image (which I know is also CentOS based) so that the host only includes the virtualization and management related functionality/packages.

I will admit I don't have a whole lot of experience with bare-hypervisor (or bare-like) setups, outside of a little bit of time in RHV at my last employer.

Unraid can do dockers and VMs

Should I be using CoreOS for my containers?

Generally I've been using either Ubuntu or Alpine as bases.

Also it looks to be Red Hat based? Is there a Debian equivalent?

I think CoreOS is more intended for use as a container host - so you'd run CoreOS on a VM or bare metal, and that machine would run your Ubuntu/Alpine containers.

Ah I see. I totally missed that.

So that's quite interesting. Because on the host I generally run Ubuntu 18.04. Because I know how it works and can ensure it's secured and up to date.

Same with someone with familiarity with Redhat/Centos etc.

But installing a whole different OS?

Like I get it on niche Linux distros where people like packaging their desktop environments whilst pulling in Debian packages for everything else, for example.

But what's the advantage here?

In the OpenStack world you can use the old Fedora atomic and container Linux to automatically spin up k8s clusters using OpenStack heat and OpenStack magnum.

There are a few advantages.

First is minimalism - it's designed to run containers.

Next is automatic, atomic updates - layers that comprise the entire OS are "pulled" and updated when the host boots, similar to how updates work with containers. Because updates are atomic, they can also be rolled back.

I think there is also built-in support for rolling updates across a fleet of host machines.

I find CoreOS a very interesting proposition, but because it's so different from other distros, there is going to be a learning curve.

Oh, I forgot one! The OS is read-only while the host is up, which is a big security and stability benefit.

Fedora CoreOS is read-only?

CoreOS Container Linux is not. But changes are not persistent over redeployment. That's nice for quick testing and development, but of course fatal should you do it in a production system. And of course not ideal for security.

Disclaimer: Edited for false claims. Hope, it's correct now...

Go look at /usr on Container Linux, where the actual OS lives. It’s literally mounted R/O. Binaries are symlinked from there, so it’s impossible to modify OpenSSL, etc.

How is root prevented from modifying it? (Readonly mount alone is trivial to bypass)

The entire partition is signed so the machine will detect it

dm-verity mainly

https://coreos.com/blog/verifying-os-at-runtime.html

Bit more on dm-verity as it relates to appliance and non-appliance systems. https://blog.verbum.org/2017/06/12/on-dm-verity-and-operatin...

Btrfs will support cryptographically secure hash algorithms (sha256 and blake2) starting with kernel 5.5. I wonder if this is a suitable compromise over dm-verity, which can't be updated, and more conventional file system options? When I consider various use cases, Btrfs always results in EIO rather than propagating bad data to user space; compress=zstd:1 reduces writes, saves storage space, and can improve performance of slower storage.

There's also two interesting read-only options: read-only snapshots and read-only volume (via the seed flag), root can't write to either. Root would need to unset the flag first. A read-only seed can support writes via a volatile 2nd device, e.g. /dev/zram device, reboot and you get a reset. Or persistence via a partition. Either way a reset also resets filesystem state.

I'm shipping a test appliance out this week but we haven't completely locked on a production OS. This feels like an interesting path. Host level data persistence isn't something I need or desire, so a RO filesystem is perfect. For $reasons I've been on an Ubuntu track (changed from Debian because of an official upstream LTS). My hesitation about Fedora is the fact that it's bleeding edge (ish) and doesn't have an LTS path (reasonably, bases in the design goal). The evidence is in they fact that the upgrade path is not expected to be smooth here.

I can eat a lot of shakeup in my own data center or cloud but a shipped box makes that a lot more problematic.

> I find CoreOS a very interesting proposition

I personally find CoreOS very confusing. None of the advantages you mentioned are a problem nowadays, for personal computing. I haven't heard of Debian, Ubuntu, or RHEL update that would bork the system in a long while. All of these distros are more than capable of running containers...

And, for corporate use? You'd use Debian + Kubernetes to take care of your needs, wouldn't you?

As I said, I'm left here scratching my head, thinking up usecases for this system.

> I haven't heard of Debian, Ubuntu, or RHEL update that would bork the system in a long while. All of these distros are more than capable of running containers.

Fedora CoreOS is designed to have the smallest possible attack surface out of all operating systems currently capable of hosting containers.

Unlike Ubuntu, Debian, and CentOS, Fedora CoreOS doesn't contain packages that aren't required for hosting containers, and is automatically kept up-do-date without any manual interventions.

Just like Chromium OS on PCs, Fedora CoreOS on servers eliminates the need for "dist-upgrade", and thus reduces risks and increases reliability.

> And, for corporate use? You'd use Debian + Kubernetes to take care of your needs, wouldn't you?

CoreOS is designed for these workloads. For example, with OpenShift 4 (Red Hat enterprise Kubernetes) the masters run on RHCoreOS and the worker nodes have the option of running on RHEL or RHCOS. You can end up having your entire Kube platform running entirely on CoreOS, either bare-metal or virtualized.

Fedora CoreOS is big as OCP uses RHCOS. OKD (the community version) needed its own platform and FCOS is that platform.

CoreOS is designed to remove everything you don't strictly need to run containers and to allow for automatic-upgrades across multiple machines (ie: don't reboot everything at once). That means you have a minimal amount of things that can be hacked and easy automatic updates for the rest.

The old Emdebian project was an attempt to provide an "atomic", "immutable" model for Debian, usually for embedded workloads as opposed to server container hosting, but that's practically dead and I'm not sure that a modern equivalent is out there.

Just to reiterate what others have said, because this comes up quite a bit. You should not use CoreOS or Flatcar images as the base for your containers. They are intended to be the host operating system upon which you run the containers. Their key features that make them an awesome host OS for containers (no package manager, for example) make them unsuitable for use as the base image of containers.

Still no okd 4.3 though. Waiting

This. I was waiting for new CoreOS just because of being the blocker for OKD release.

Still trying to land the remaining changes to 4.4. It’s working, just a lot of workarounds still waiting to be replaced with real changes. It has definitely taken longer than anticipated to line up all of the impact from Fedora CoreOS cleanup and changes.

OKD Preview has been released in November (uses Fedora CoreOS btw) - https://github.com/openshift/okd#getting-started

Reply on news.ycombinator.com