CentOS Stream, or Debian?
jonathancarter.org
jonathancarter.org
We use CentOS at the office for obvious reasons but, my personal servers are always Debian since 2005 or so. Had no problems whatsoever.
Debian Testing is also a great, semi-rolling desktop distro. May be not the latest and greatest but, it's very very dependable.
(a little more insight is here: https://news.ycombinator.com/item?id=25349371)
A lot of the ML/DL stuff is developed on Ubuntu first, causing occasional headaches when deploying on CentOS, similar to traditional HPC except the other way around. Oh well..
As ML gets more popular, and now this, maybe there's a chance of breaking the RHEL (rebuild) monopoly in HPC.
Seeing Ubuntu on the scene is nice. From what can we see, our users can run their ML/DL loads relatively effortlessly on CentOS for some time.
Moreover, with GPU support on Docker, we guess that people will migrate to containers to run their loads so distribution will become irrelevant in the future.
Do you use IB on your nodes? If yes, is it easy to deploy the kernel modules?
Yes, containers are a god-send for those apps that require some specific version of a specific distro.
> Do you use IB on your nodes? If yes, is it easy to deploy the kernel modules?
Yes, we do. We use Mellanox OFED, which supports Ubuntu, so it's no more difficult than on CentOS. Perhaps even slightly easier, since minor Ubuntu version updates don't break the kernel internal ABI's so there's no need to wait for all your out-of-tree kernel stuff (MOFED, Lustre, whatnot) to be updated before you can upgrade.
- Software is released RPM first.
- Software certification done on RedHat and CentOS (compile / verification).
- Library versions present in RedHat and CentOS are used in development.
- Documentation is written RedHat first.
You can do everything on Debian. Possibly something would be hardcoded and you need to work around or re-compile it. Or, it works without hiccup but, developers can't provide support since they didn't test on Debian.Scientific software is a very different beast. Its accuracy and precision sometimes depend on the mood of a newborn butterfly in Zanzibar. I develop such software so, I know a little bit about the peculiarities.
Since RedHat also has paid support, RH based distributions are considered more sound and mature, because if something breaks, RedHat can fix. It's no different for Debian but, RedHat was able to fly an engineer for NEC Corp. to reproduce and fix a GCC bug on-site. System in question was Itanium based IIRC.
I'm sure you can get this stuff working on Debian. I'm also sure it is much more of a PITA to do so, just because that's not where the eyeballs are.
Security fixes are for 5 years.
https://wiki.debian.org/LTS/Team https://wiki.debian.org/LTS/Funding https://www.freexian.com/services/debian-lts.html
The LTS project supports most packages and the unsupported subset is specified, with the debian-security-support tool able to report unsupported packages that are installed:
https://salsa.debian.org/debian/debian-security-support/-/bl... https://packages.debian.org/stable/debian-security-support
https://wiki.debian.org/LTS/Extended https://deb.freexian.com/extended-lts/
Edit: it's "devel" not "develop"
To keep on rolling, you can use the keywords (stable, testing, unstable) in sources or use release names to stop rolling when something releases (buster, stretch, etc.).
The nice thing about using release names is the annealing effect during development. First major version bumps stop (feature freeze). Then minor version bumps stop (RC state / full freeze). Then one day you read that Debian is released. Congrats! A last apt-get and you've arrived in stable.
I always deploy current stable to any production server and I keep my desktop systems in rolling testing.
What's interesting is that close to a freeze, "testing" is quite stable already, so pinning to its name (so bullseye as of now) allows you to keep this rolling state until it converges to the next stable.
You can manage these suites in two different modes: rolling (suite based) or non-rolling (codename based)[1].
As you also noted, testing has a sinusoidal rolling speed. It starts rolling slowly when a stable is released. Then rolling gets faster. Then slows down (feature freeze) and stops (RC/Full freeze). Then that version becomes stable. Testing separates and whole cycle starts again.
Testing never becomes unusable. Unstable has this risk. Instead testing is always tried to be kept in a semi-rc state. Any breaking bugs is a barrier for unstable to testing migration. As a result, testing can be used as a desktop system without any big drawbacks.
[1]: Since unstable is always codenamed sid and experimental has no codename, they always roll. Similarly since stable is rolled from release to release, it doesn't quite roll but backports and volatile allows it to creep forward in a controlled manner.
1) by explicitly referring to "testing" in your apt configuration. Then it's rolling forever.
2) by referring to the current "testing", so for example "bullseye". Then you're temporarily on testing, rolling for a while until the day bullseye becomes the new stable, and then you're on stable too. I often do this a bit ahead of a new stable on my personal laptop, to check what's coming with the new stable. I guess it's probably quite common.
You can always stay on devel, or switch to a stable release when (or preferably few days before) it's released.
Had to reinstall my 8 year old installation to switch to 64 bit, and never reinstalled the new one.
So you basically have this ordering more or less in terms of stability and how dated the packages are:
From older (and more tested) to newer (and less tested): Debian Stable | Ubuntu | Ubuntu devel <-> Debian Testing | Debian Unstable
Ubuntu devel <-> Debian Testing meaning that their orders depend on their respective phases, e.g. Debian Testing will soon freeze (prior to the next stable release) so in that time Ubuntu devel will probably be more up-to-date by virtue of being based on unstable. The rest of the time there's automatic migration from Unstable to Testing provided no blocking bug is discovered, so I'd expect packages in Debian Testing to be more recent.
Honestly I think the main differences are the default experience, but if you know which packages you want to use, Debian and Ubuntu should be pretty similar.
What I have realized in these years is that its stable branch is extremely stable. I have never ever faced any issue with dependency management or package installation with the stable branch. Yes, the packages can be sometimes a couple of years old but the whole system is exceptionally stable. I believe it sets a benchmark for stability.
A couple of niggles so far 1) nvidea drivers broke, had to install the latest version with apt 2) Sound wasn't working from user accounts, pulseaudio complaining about unable to access the files. lsof showed timidity had snd files open too, and removing that seemed to do the job 3) clusterssh reported a perl issue - for some reason I had a ~/perl5 folder in my home directory
vlc looks weird now too, but overall not too bad, and will last another 4 years. Not sure if this desktop will last beyond 2024 - it's a 2015 HP Z440 with a Quadro K2200 graphics card.
Over time I've seen bad things with ubuntu -- changing networking, breaking logging by forcing systemd, etc. Ultimately though you can only delay change, but like Cnut, there's no much you can do about the incoming tide. Having breaking changes every 4 or 5 years is the price to be paid.
Server wise we never upgrade, reboot, pxe boot, build the latest, then run the installation commands. Had various issues over the years with updating internal packages (apache 2.2 to 2.4 changes come to mind)
It has sane defaults (if you are OK with Gnome) and easy installation of (non-free) nVidia drivers / CUDA / tensorflow. Steam works out of the box.
It's as close as any OS (Mac/Windows) or other linux distro I've used to being ready to go from the first login, and staying that way with minimal fuss.
It is a distro that feels like a full OS instead of a random conglomeration. My only negative with it is it uses Systemd in general, and Systemd boot in particular. If you are fine with that, it is just about the perfect distro.
(I prefer open-rc as an init system/service manager, I recognize that Systemd makes sense for Pop.)
First of all, the general transition to Linux was started by engineers, customers of UNIX, who were tinkering with Linux. We weren’t software engineers and not super interested in how Linux worked, just what we could do with it. Debian was a chore to install, in the way of getting things done. Red Hat, on the other hand, was easy to install. Red Hat was the distro of choice for experimenting with Linux.
The other side of it, the business side, is that RHEL met the need of, “see boss, this is fully supported, just like UNIX.” That had to be a thing before these types of businesses would transition from UNIX to Linux. Only after that transition would front-line engineers be brave enough to sneak a RHEL clone like Centos into the mix. Sadly, Debian just didn’t stand a chance in this situation.
Fast forward to today and now you see front-line engineers hacking Ubuntu LTS distros to look enough like RHEL for the commercial tools to run on it. You still have to keep a RHEL box around to reproduce issues before going to support with issues in the commercial tools, but that’s much cheaper than a whole fleet of RHEL boxes.
Debian was a relative chore to install in the pre debian-installer days. But d-i has been a feature of Debian since Sarge (v3.1, released in 2005) and it brings Debian to parity with other distros.
$prevjobcompany lives by that unwritten rule for more than two very profitable decades by now, and I recommend anyone considering operating a Linux-based OS to adopt it.
There are glacial changes, but scientific instruments are often still shy on Debian or even Ubuntu :(
It can't. Not just drivers, firmware updates for BIOS among other things. Ubuntu is getting there but it isn't considered as seriously by all manufacturers.
There's a growing distro-agnostic solution here, LVFS fwupd https://fwupd.org/
We're one of those (who uses, not who manufactures)! But in most cases it is not a question of Linux distribution and more of is it Windows (hopefully not XP) or not :(
So all updates and bugfixes will be pushed to CentOS Stream first, but they'll go out of their way to circumvent this for security patches. Just because of their paying customers.
Makes sense but it's kinda funny.
The OP is mostly about being a Debian fanboy.
It doesn't even consider that Fedora uses pretty much the same level of packages that Debian has and offers stable release upgrades for 6 years now.
Because Fedora has no such thing as LTS. But is that really necessary if you treat it like a rolling release?
I'm of course a RHEL fanboy and I love stuff like SElinux and yum/dnf too much to go back to Debian.
(Though I mostly use OpenBSD now -- when I have to use a linux, it is debian or devuan.)
Edit: also, Debian is very easy to install, has extensive documentation, and (IIRC) commercial support is available. There are reasons that there are more eventual Debian derivatives than of any other distro.
If that’s what you’d need, then why not just run a Linux distro to start with?
Now however with WSL2/Docker enabling easier development I'm tempted to take taco-bell style shortcuts, it'll go quicker in many ways but also reduce portability to other platforms like FreeBSD (that while a favourite overall ends up behind in total if i start taking these shortcuts).
From personal experience, I've broken way more ubuntu installs on a release-upgrade than I have on debian (zero).
https://bugs.launchpad.net/ubuntu/+source/base-files/+bug/18...
First thing /etc/update-motd.d/50-motd-news does is source /etc/default/motd-news if it exists
Only if that file sets "ENABLED" to 1 will it do anything
[ "$ENABLED" = "1" ] || exit 0
I haven't knowingly deleted that file, not sure how it opts in. Perhaps this line in my preseed removes popularity-contest popularity-contest/participate boolean false
Notably though, with base-files 10.1ubuntu2.10 and $ dpkg -L base-files | grep motd-news
/etc/update-motd.d/50-motd-news
/lib/systemd/system/motd-news.service
/lib/systemd/system/motd-news.timer
There's a mention on the prerm about removing that conffile.Also, Ubuntu adds some bloated software of dubious quality.
Why Ubuntu instead of Debian?
It's an everchanging, relatively buggy project, without upgrade support, and full of those weird choices (unity, phoning home) it tries to impose onto every user.
The main difference is that in Ubuntu the fix usually involves removing some program that is messing with everything. While in Debian it usually involves adding some program that isn't installed by default for some reason.
(Debian would gain a lot by making an installer with the backports kernel, by the way.)
Despite that, at the company I'm helping to found as CTO, I may need to set a default of Windows in the short term: the story for using MS Office and screen sharing / other fancy features of Zoom on Linux isn't great, web dev on Windows is getting surprisingly good with things like WSL and VS Code, we don't already have a fleet of Intel-based Macs to use, and for now the new M1-based Macs are likely to demand workarounds for the CPU architecture on which we can't spend the necessary time at this extremely early stage of our corporate lifecycle.
Macs will probably become a viable option for us again in a couple of years, and Debian / Ubuntu will probably be viable from day 1 for any technical employees who don't need the commercial software I listed above.
Ubuntu today is much less opinionated than before in desktop use, since they switched from Unity to the same Gnome that most other distributions are using.
sudo apt-get remove snapd
sudo apt-get remove lxd-agent-loader # 20.04
The key to keeping them uninstalled is to pin the packages so they don't get "upgraded," which results in them being reinstalled. sudo apt-mark hold snapd
sudo apt-mark hold lxd-agent-loader # 20.04
If you want to go the next step in cleanup: sudo umount /var/lib/snapd/snaps/*.snap
sudo rm -rvf /var/lib/snapd/*
sudo rm -vf /etc/systemd/system/snap-*.mount
sudo rm -vf /etc/systemd/system/snap-*.service
sudo rm -vf /etc/systemd/system/multi-user.target.wants/snap-*.mount
sudo rm -vrf /snap/*https://packages.debian.org/search?keywords=linux%2Dimage%2D...
Backports can be activated by adding one line to your sources.
If I trust https://www.kernel.org/, the last stable kernel version is 5.9.13, and according to https://kernelnewbies.org/LinuxChanges, 5.9 has been released two months ago.
There are plenty of reasons to criticize Debian, but that's not one of them.
But if you need a stable and secure server Debian or Ubuntu are a joke. There are no engineers with Debian, only lawyers and community managers. There your can use RHEL, Oracle Linux (with dTrace and ZFS), or SLES. Those do have the experts. For an unstable server CentOS Stream is fine.