OSTree – Robust OS upgrades for Linux
blog.verbum.org
blog.verbum.org
This is the part where I suggest docker people broaden their scope to include such... "devopsy" concerns around virt deployment. In my own system used internally within my company we have this distinction .. the platforms are called PEs (platform environments) and the applications are SEs (service environments). Combining both produces a SIN (service instance node) on a particular CP (cloud provider).
While I really support docker and they probably get annoyed at me always commenting on their project and coming across as being slightly critical, in all honesty docker irritates me because it leaves all this sort of business out of scope. But I think they are possibly also heading in this direction. :)
Basically, it seems like CoreOS might replace Ubuntu as the default host environment for the docker userbase, however the issue of supporting more exotic environments is apparently not being tackled.
Vagrant is just one of the options for running CoreOS that lots of people like because it is easy to test out. It isn't the only option though. What do you mean by paravirt?
My comment was about running arbitrary things on CoreOS, not the other way around. (Sorry if perhaps I wasn't clear enough, IMHO it goes without saying that Linux will run on any standard platform.)
Huh? I thought the point of vagrant was to just boot up a known image. Then you have veewee which can build an image suitable for vagrant from kickstart or whatever. Can you elaborate on your concerns? Thanks!
Provisioners in Vagrant allow you to automatically install software, alter configurations, and more on the machine as part of the vagrant up process.
This is useful since boxes typically aren't built perfectly for your use case. Of course, if you want to just use vagrant ssh and install the software by hand, that works. But by using the provisioning systems built-in to Vagrant, it automates the process so that it is repeatable.
It's the last part of the final sentence. You can't prove something is repeatable if it's potentially accessing the internet, date and time logic, etc. Cloning an image is a lot more repeatable (and known quantity) than re-massaging one in to existence.
I term the class of IMHO misdirected systems administration tools that attempt the latter PFCTs (post-facto configuration tinkerers). Vagrant is on the better end of these (at least it focuses on one-time instantiation versus constant modification). Other tools such as puppet could be construed as encouraging long term modification without a clean reference at all, thus potentially resulting in configuration drift.
It seems clear to me that this class of tool grew somewhat organically from classical systems administration and is not the most rigorous of approaches in this era of cheap virtualization and instant cloud-based provisioning.
I think there's been some editorialising of the title as I don't see that the author of OSTree claims that it's a "robust" solution (alone).
With that caveat, without filesystem snapshot support, OSTree really isn't a complete solution (as the original author points out).
On Solaris 11+, there are generally two update scenarios for package upgrades:
1) pkg update [name1 name2 ...]
no packages have new or updated items tagged with
reboot-needed=true
pkg will create a zfs snapshot
pkg will create a backup boot environment
perform update in place on live root
if update fails, will exit and tell admin name
of snapshot so they can revert to it if desired;
will also print name of backup BE
if update succeeds, will destroy snapshot and exit
one or more packages have new or updated items tagged
with reboot-needed=true
pkg will create a zfs snapshot
pkg will clone the live boot environment
pkg will perform update on clone
if update fails, will tell admin name
of clone BE so they can inspect it and exit
if update succeeds, will activate clone BE,
destroy snapshot, and exit telling admin name
of new clone
How do you determine reboot-needed=true? Is that something assigned by the package developer statically (i.e. kernel is reboot-needed=true)? Determined dynamically at update time (like in the package metadata for a specific revision?)
Do you attempt to control for local configuration?
Is the X server package reboot-needed=true?
If you think OSTree needs filesystem snapshot support, you
aren't understanding how it works. Or perhaps we don't
have a shared definition of "robust" - for me, assuming
correct behaviour at the filesystem and block layer, I
believe deployment changes to be atomic.
Also, I'd argue that assuming correct operation at the filesystem and block layer is a pretty big assumption :-) Even if the software itself works correctly, time has proven that trusting the hardware is not a good thing.
With that said, yes, I think OSTree needs filesystem snapshot support if admins are to rely on it to provide "robust" OS upgrades.
How do you determine reboot-needed=true? Is that
something assigned by the package developer statically
(i.e. kernel is reboot-needed=true)? Determined
dynamically at update time (like in the package metadata
for a specific revision?)
Further refinement of this functionality is planned when applied to specific types of actions found in a package manifest, such as "driver" actions. But it will be handled by the package system transparently.
Do you attempt to control for local configuration?
If you're talking about service configuration, that's handled via SMF (the service management facility) which has its own snapshot system that services use.
If you're talking about legacy service configuration that uses flat files on-disk, then package creators have the option of using service actuators to trigger a restart or refresh of the service whenever the configuration file changes.
You'd have to be more specific.
Is the X server package reboot-needed=true?
An incompatible change in the X server itself (such as a protocol change, which is very rare) will likely come with a related kernel change which would require a reboot. But again, the package creator is in control of that.
Also, the "reboot-neededed" tag doesn't force a reboot per se; it just forces the update of a given component to be performed in a clone boot environment.
I'm happy with the fact that it allows admins to choose whatever they want at the filesystem and block level, the same as dpkg/rpm do. For example in a cloud environment, block level redundancy may be more easily provided at the infrastructure level for guests.
For those who do need redundancy, you're free to choose BTRFS, XFS+LVM, or hardware raid, whatever suits you.
As for the rest of your reply around the reboot-needed flag; it basically sounds like it's fairly manual, but maybe that's "good enough". I have a particular paranoia about race conditions though, so were I to design a system that attempted to do live update applications, it'd be a whilelist, not a blacklist as reboot-needed effectively is.
I'm happy with the fact that it allows admins to choose
whatever they want at the filesystem and block level, the
same as dpkg/rpm do. For example in a cloud environment,
block level redundancy may be more easily provided at the
infrastructure level for guests.
For those who do need redundancy, you're free to choos
BTRFS, XFS+LVM, or hardware raid, whatever suits you.
As for the rest of your reply around the reboot-needed
flag; it basically sounds like it's fairly manual, but
maybe that's "good enough".
As for actually being manual, not really. The package system provides package creators with a tool called pkgmogrify, which allows them to set rules that cause transformation of actions based on patterns at package publication time. As an example, Solaris has a set of rules that say that, by default, any files delivered to /kernel should be tagged with reboot-needed=true.
I won't claim it gets the OS 100% coverage, but as has been said before "perfect is the enemy of good."
I have a particular paranoia about race conditions though,
so were I to design a system that attempted to do live
update applications, it'd be a whilelist, not a blacklist
as reboot-needed effectively is.
For example, on my desktop workstation, ~289,806 items have been installed by the package system. Of those, only 857 have been determined to require a reboot if they are updated.
That means that roughly 0.37% (yes, < 1%) of the items on my system have been determined to require updates to be performed in a clone boot environment (require a reboot).
For enterprise-level customers, minimising the number of reboots needed to enact change is paramount. Any downtime at all can often cost them millions of dollars.
If you think OSTree needs filesystem snapshot support, you
aren't understanding how it works. Or perhaps we don't
have a shared definition of "robust" - for me, assuming
correct behaviour at the filesystem and block layer, I
believe deployment changes to be atomic.
If the answer is that an administrator must define what StateSRC is every time they upgrade their system, I don't think that's a usable mechanism.
If it does it based on the live system (the best answer), then without filesystem-level snapshots, how can OSTree reliably define StateSRC? There's a non-trivial set of race conditions OSTree would encounter while attempting to create its own "snapshot" of the state of the system.
Having filesystem-level snapshots allows you to get an accurate view of the live state of the system so that you can then modify a copy.
With ZFS, snapshots are basically "zero cost". Or rather, they have very low overhead for the base snapshot, and you only incur additional expense from divergences in the snapshot from its parent.
Your comment strongly implies to me you don't understand how it works. It's really really simple - the physical / of the filesystem no longer stores the OS. Instead, you have multiple chroots in /ostree/deploy/osname/{checksum1, checksum2} etc.
To make a new checkout, it's just a hardlink farm.
This kind of thing has been done to some degree before - it's not entirely new. OSTree just polishes it, makes it more efficient, documents the semantics of /var, etc.
How do you define the initial chroot? Who is responsible for creating that?
How would you migrate an existing system to use OSTree?
If an administrator has to manually create the initial chroot link farm, I don't think that's a very usable solution. In particular, it suggests a lot of process overhead that assumes most systems are "templatable" representations.
It's not immediately clear whether OSTree deals with necessary in-place upgrades or if it only helps you with reboot-based ones.
Even if you don't feel in-place upgrades are safe, again, most enterprise-level customers are demanding update solutions that don't require them. There's a reason why ksplice, et al. are becoming popular.
1) updater makes a private snapshot of the filesystem
2) updater writes its changes to the private snapshot while the rest of the system gets served by unchanged FS
3) updater stops affected services
4) the updated snapshot is swapped in (atomically)
5) services are restarted
Designing applications/services for systems with read-only root filesystems is a good way to enforce known location of on disk state.
The above is the strategy I am taking with a personal project used internally by my company which predates docker, works for arbitrary OSs and has far wider scope.
With regards to block data, I took a different approach to docker, namely using LVM2 LVs instead of aufs. The limitation of LVM2 LVs is snapshot is possible, but not yet snapshot-of-snapshot. That means whatever you want to spin up either has to be spun up slowly (ie. not from a snapshot) or has to be non-snapshottable when running (due to LVM2's snapshot-of-snapshot limitation). We opt for the latter, which requires that state is not stored within the snapshot-powered system. This is good because you can power them off without fear, like conventional PXE-style diskless systems.
That's precisely why a manual upgrade process should not exist for system environments. Instead of live upgrades, you name, version, and test a newer and known-good image of the environment with the applications required on it on test infrastructure prior to final staging/deployment.
Instead of rebooting machines into a new filesystem, docker spawns processes directly into them, using the sandboxing capabilities of the linux kernel. So you get a much more powerful and flexible deployment mechanism. But of course, you need a machine to exist in the first place, which docker isn't designed to do.
So OSTree is a good companion for Docker, because you need a machine to run docker. A good workflow is probably: pack the bare minimum on a physical machine, using ostree. Then put everything else into docker containers. That's the approach of new "just enough" distros like CoreOS.
One fundamental difference is that OSTree has a custom serialization format for trees (inspired by git), whereas from what I can tell from the code, docker is just tar (I assume whatever the host /usr/bin/tar serializes to). For example, OSTree explicitly supports extended attributes, so it can support SELinux (and SMACK). Fedora ships a patched tar but...the tar format is a really serious mess.
I would further add though that OSTree does, providing the OS is compatible with it, allow booting a separate "deployment" as a container. So for example if you have Debian in /ostree/deploy/debian/90cd266 while you're booted into /ostree/deploy/fedora/562d0a, you can easily just systemd-nspawn /ostree/deploy/debian/90cd266 and boot the same OS as a container.
But the emphasis right now of OSTree is indeed on bare metal deployments, and I'd like to push hard to integrate with package systems.
Perhaps we should be looking at integrating ostree and docker then? :)
> I'd like to push hard to integrate with package systems.
How would that integration work exactly?
It might make sense for docker to be able to store containers as OSTree commits in addition to tarfiles. But I haven't used it myself. On the HTTP side, OSTree may be less efficient or more efficient than what docker does on the wire for updates; I don't know. Static deltas will help significantly.
> How would that integration work exactly?
This section describes it very briefly: https://people.gnome.org/~walters/ostree/doc/adapting-packag...
Basically this is something you can do on a build server or on a client.
> OSTree only supports recording and deploying complete (bootable) filesystem trees. It has no built-in knowledge of how a given filesystem tree was generated or the origin of individual files, or dependencies, descriptions of individual components.
Um-m, isn't this what snapshots (e.g. LVM snapshots) are for? Correct me if I'm wrong.
See also the Solaris 10 IPS, which appears on the surface to have many similarities (checksums, http delivery of files, etc.) to OStree
But of course, all the cool new features in Linux have been in use in production by Solaris people for years. :-)
Docker works in a really similar way, actually -- it creates file system images and tars them up, and then uses AUFS to separate changes in each container from the base image. Docker added metadata to describe how to build images; if you want to change the components (AFAICT) the "right" way is to change the metadata and then do a full rebuild.
Correct. And Docker implements caching of build layers, so you get the semantics of a full rebuild, but in practice are only rebuilding the interesting parts.
I will try to explain the concept less accurately. For example, you have your favorite os versioned in git. (kernel, x server, apps, init daemon, session management...). This tool will take all the stuff, compile it, will create directory structure and place binaries to appropriate places. In the end it will create bootable directory structure (filesystem tree) to that you can chroot. But maybe you want to test different revision of systemd. So ostree will take chosen revision from git, compile it, add binaries to filesystem while preserve old structure. It will add only new things that are different. Both verisons will be bootable. It's basically binary mapping of git tree.
So...what you end up with is a bit of a layering mess because you have to keep track of both block and filesystem level bits.
While OSTree does pay a penalty in complexity in other ways (mainly the chains of indirect symbolic links for boot=), it's extremely flexible in that you can do whatever you want at the FS and block layer.
You can absolutely use LVM snapshots and OSTree together, and that's very much intended. For example, in the OSTree model, if one object gets corrupted, all trees using that object are corrupted.
So it's very much still recommended with OSTree to add redundancy at the block layer with LVM/RAID/BTRFS - again, you can do whatever you want there.
Finally, LVM snapshots don't directly help you parallel install independent operating systems with their own /var, and optionally /home.