I don't understand why people use Gentoo in production servers. Is it good practice? I can see so many downsides.
In practice, many people don't understand that Gentoo is a meta-distribution and try to run it like a simple distribution, leading to arbitrary ABI breakage and chaos. If you want to run Gentoo on servers, you need to test all upgrades on build machines before rolling them out. You need to publish your own binary packages. You need to do a lot of the testing that Debian, Ubuntu, Red Hat, etc. would be doing for you.
I have some friends who set up their own hosting business based on Gentoo and they seem to be doing it right. Their servers are very stable. Testing updates is an ongoing expense, but I think it has worked out well for them.
> Compared with Debian or Red Hat, Gentoo is really a meta-distribution. It's a nice basis for creating your own distribution.
I do not want that on my production server. We're in the $SOME_BACKEND_SERVICE business, not in the distro creating business.
> In theory, Gentoo lets you mix the latest version of some packages with well-tested, old versions of other packages and generate a release perfected for your organization.
I want a hyper stable base that is tested by the masses and when possible has some big org behind it who's supporting it (and thus very conservative when changing stuff). On top of that base I might roll my own packages or include a fringy package repository. Never I want the base to be different for everyone.
> If the Gentoo release of some package, Ceph for example, isn't as up to date as you want, you can easily update it yourself and publish a new release to your servers.
This is not too hard for any distro I've used.
> In practice, many people don't understand that Gentoo is a meta-distribution and try to run it like a simple distribution, leading to arbitrary ABI breakage and chaos. If you want to run Gentoo on servers, you need to test all upgrades on build machines before rolling them out. You need to publish your own binary packages. You need to do a lot of the testing that Debian, Ubuntu, Red Hat, etc. would be doing for you.
So may we conclude, as GP said, that running Gentoo on production servers is just not a good idea?
> I have some friends who set up their own hosting business based on Gentoo and they seem to be doing it right. Their servers are very stable.
This is a little too anecdotal for me :)
I've done my share of Gentoo, and this system in intriguing. But when I need stability, as I need for backend services, then it will not even be on the long list.
I found some similar-level intriguing features in NixOS. I'd probably still not run it for backend services, but I might be compelled to run it on the application layer in some cases.
But honestly, I think most people who run Gentoo do it for the learning experience and for fun. :-)
Yeah, nice one! Very customized flags required.
I was more talking about typical backend services, like Ceph or, say, a Riak or Postgres cluster.
Minimise your deviation from the stable stuff that everyone else is using.
But still, maybe there is some stuff with special processors that needs all to be compiled specifically with support for that type of thing, in order to make proper use of the underlying machine.
> I do not want that on my production server. We're in the $SOME_BACKEND_SERVICE business, not in the distro creating business.
A lot of companies know that Google and Facebook have their own distros. Ergo, to become as successful, they too must build their own distro.
This hasn't stopped Google from using a Debian variant internally as a desktop OS, so they all kind of have their place depending on what you intend to build.
I don't at all read your parent as asserting that all production deployments should make use of a meta-distribution like Gentoo, but instead describing the specialized features that are useful for some specific niches. If you have some business reason to want to run arbitrary combinations of different versions of different software together, or to more-easily manage distributing software to your platform with many different features that you sometimes want compiled out, or otherwise if a slow stable extremely-reliable distribution isn't handling your needs, you might want to consider using Gentoo instead of building all that infrastructure yourself.
That's not the case for you, or for me, and that's fine. You've described some of the reasons I also prefer a stable well-supported distribution and don't use Gentoo. My reading of your comment is saying that because you personally have no use for the benefits, therefore Gentoo is completely unsuitable for any production use ever. If I've misread your comment, and you were just commenting on your personal use cases without trying to speak to any general audience, I apologize for my mischaracterization.
Portage is a great tool, but I always thought it was better suited to build packages and operating systems as opposed being an end user package management solution.
Reproducbility is still a difficult problem, but that isn't one of them, and Nix comes far closer than Gentoo.
As for reproducibility when building from source, that depends on your practices. Is every box a special snowflake with it's own USE flags and package set? Well then yes, good luck with reproducibility. Or are you standardizing your targets and building one package to deploy to them all?
It, of course, depends on the project; if the ./configure script drops the current date in source code, it'll be different, but that doesn't happen often.
Software and firmware updates from Linux distros have recently bricked modern hardware several times. This should never have happened, because a period of testing is necessary to evaluate whether a given update will cause failures. With rolling release, you update and pray.
The nice thing about Arch, its absolutely trivial to rollback a package, or even rollback all your packages to the ones published on any given date. So I was able to deal with all of these issues quite effectively which is a credit to Arch, but it simply is the case that you will see more breakage on the leading edge.
What you're describing there is bleeding edge; not rolling release. If you ran a Ubuntu with the testing repos then you'd have experienced the same issues.
> The nice thing about Arch, its absolutely trivial to rollback a package, or even rollback all your packages to the ones published on any given date. So I was able to deal with all of these issues quite effectively which is a credit to Arch, but it simply is the case that you will see more breakage on the [bleeding] edge.
Indeed you will on bleeding edge. That was my point. Most of the complaints people attribute to rolling release are actually problems with bleeding edge distros rather than rolling release. You've downvoted me only to reiterate the same point.
I'd covered that in my first post as well (did you even read it? :P)
> without a release cycle, what exactly would they be waiting for?
Rolling release distros still have release cycles. eg they often have testing repos where many packages will be trialled before they hit the main repositories. Much like you see happen with packages which get updated between major releases on non-rolling release distros.
The difference between rolling release and non-rolling release is a bit less pronounced these days because even most non-rolling release distros now have easy upgrade paths from one major release version to the next. Heck with the Debian / Ubuntu derived distros you can just update your apt config to point to the next release repos and apt will carry on as if you're a rolling release distro. So I think the real difference between rolling release and non-rolling is simply just that you're given greater assurances[1] that you don't need to perform manual intervention during package upgrades outside of the major version upgrades where as with rolling release the risk of manual intervention can come without warning[2]
In terms of package stability, there's nothing stopping someone creating an Arch fork which runs packages a few months behind the main Arch repos. In fact I think that might have been the concept behind Arch Server - not that it ever took off.
So anyway, back to my original point: rolling release doesn't have to mean "unstable". It just means breaking changes don't all get held back until major milestones are reached. But that's often true for non-rolling release as well. It just so happens that most rolling release distros are also bleeding edge - and that is where the trope come from. So saying rolling release can only be unstable is akin to saying Redhat can only be bleeding edge if you only ever use Fedora.
[1] I was going to say "guaranteed" but that's not quite true either; eg FreeBSD will happily perform breaking changes to applications (within a major release version) if you're not careful what application versions are being installed
[2] This is also not quite true but many distros will warn you about breaking changes in their news feed. But also the package manager itself gives strong clues too (eg Apache 2.2.x -> 2.4.x will obviously result in manually updating some Apache config files).
No, you should have build hosts for that, publish binpkgs, and then install the binpkgs.
Gentoo's binpkgs do not require compiling on your production machines; compile them elsewhere.
> an update might require configuration file change, this doesn't happen in CentOS or Debian Stable
Debian has debconf, and Centos has a similar tool. When configuration files change in the debian world, they often prompt with those maintainer scripts.
Gentoo's dispatch-conf/etc-update model is quite nice because it also lets maintainers recommend new config defaults which the user may easily accept or reject.
You clearly don't know what you're talking about. Then again, many people who use gentoo don't use it as you would in production, with binpkg mirrors and well managed package unmasking/masking/etc.