Hacker News Reader

I have a ton of public-facing servers on the Internet, running RHEL, Debian, Ubuntu, FreeBSD, and, yes, there's even some Windows machines in our datacenters (fortunately, I don't have to touch those and haven't logged into a Windows box in years). My personal machines (at home and in the datacenter) run RHEL, FreeBSD, OpenBSD, and Debian and my work laptop/workstation runs Ubuntu. My e-mail ends in @gnu.org and I don't work for a "Big Enterprise, with Big Serious Enterprisey Stuff". I work for a "Small-ish ISP, with Small Not-Serious un-Enterprisey Stuff" that ranges from web and mail servers to monitoring systems to file servers to RADIUS & TACACS servers. I started with Slackware in 1996, jumped to Debian soon thereafter and used it exclusively for about 10 years before "branching out". Now, I just use the Best Tool For The Job(TM). I also avoid making blanket statements about groups of people based on their software choices.

I'll bite the Ubuntu part.

Ubuntu was the second Linux distribution I put my hands on back in 2008, after giving Fedora a try and being frustrated - it was a system administration internship at my university, every server on the campus ran Fedora and our "CTO" (and also my boss) was a heavy Fedora user. Ubuntu was so much more welcoming for a person coming from Windows.

When I began doing some serious job in the cloud (~2010) it was on AWS and apart from Amazon's own AMIs, Ubuntu was the defacto standard - I even remember reading that it was the most used distribution on AWS back then (don't know the numbers nowadays).

I've never felt that I was using something under engineered, bloated or big. It simply worked very well, had fairly updated packages and a 5 years support that has always made sense to the tasks and jobs I've dealt with.

I like to think that despite some questionable choices, Canonical has always had a honest marketing approach with the Ubuntu brand, partnerships, embracing the cloud (juju), etc. When Vagrant came out, Hashicorp had the Ubuntu based box and now canonical maintains its own trusty box. It just always seemed to be present on every corner of my career so far, and it never let me down.

Please note that I'm not saying that Debian is not a superior choice, I just think it is unfair to look at Ubuntu with disdain and bring JavaScript and Ruby developers, as if the languages or its developers shared the characteristics you've brought. I simply never had a reason to move away from Ubuntu or try Debian.

> Ubuntu was the second Linux distribution I put my hands on back in 2008, after giving Fedora a try and being frustrated

For me it was RedHat, then Fedora, then Ubuntu, then Mint, then Debian.

> I just think it is unfair to look at Ubuntu with disdain and bring JavaScript and Ruby developers

The thing is, Ubuntu is kind of the Blub of distros: folks fond of it it can see where it's better, but not where it's worse.

As for server-side JavaScript and Ruby: I maintain that they are smells in any system. JavaScript is a mistake whose popularity relies on an historical accident. Ruby's a neat little language, but every single system I've used which utilises Ruby has been broken.

Actually, the reason I use Ubuntu instead of Debian is that most language runtimes I use (python, for starters) were, for quite a few years, more up-to-date. Of course I could use unstable, but I'd much rather use Ubuntu's LTS and security updates.

Yes, Debian will pretty much just work, but back before 7.x it would also be a pain to support a number of applications (we mostly rolled our own builds of "modern" runtimes and libraries we had to use, with predictable lag times and dead ends). Ubuntu takes all of that pain away and has relatively new stuff, even on LTS (sometimes through backports, but hey... nobody's perfect).

I also left RedHat because of the way the distro evolved, but never jumped to CentOS because at that time yum was slower than molasses, and I'd much rather use a Debian derivative anyway.

> Actually, the reason I use Ubuntu instead of Debian is that most language runtimes I use (python, for starters) were, for quite a few years, more up-to-date. Of course I could use unstable, but I'd much rather use Ubuntu's LTS and security updates.

It's almost definitely better to install stuff in /usr/local or in install-specific locations under /opt than to use either unstable or Ubuntu in production.

> Ubuntu takes all of that pain away

Yeah, but it replaces it with its own pain. The days I bid farewell to Unity and GNOME were so happy…

I don't think any of those apply to production servers (at least not how I picture it).

Ubuntu is just easy. You do apt-get install and you get the package you expect. Debian and Centos just don't have a lot of packages, that is the biggest reason for me.

Strange, this is exactly what I'm saying to my friends, but about Debian. apt-get has almos everything, which packages aren't there?

For not caring a lot about open source, they have many MANY more open source developers than Canonical does, or likely ever will have. Also, I suspect quite a few Redhat employees will take strong offense to your comments on them not caring about Free Software. Funny, Redhat never had issues such as this that Canonical did about licensing terms not being GPL friendly, which they have since fixed: https://www.fsf.org/news/canonical-updated-licensing-terms

I'll totally agree with you on Debian vs Ubuntu, but on Redhat vs Debian... We can agree to disagree. Here is a paste of an email I wrote up to some coworkers on the subject:

======================================================

RPM starts off with the concept of pristine sources. It is vehemently rejected that a maintainer of a rpm package use a non-upstream tarball or change _anything_ without a patch that is in version control. This is not the case with dpkgs

RPM

===

• Stores ownership, permissions, and checksums in the rpmdb. This allows for tools like [1] which are entirely impossible to re-create a dpkg equivalent. There is no equivalent of the insanely handy “rpm –V” in dpkg.

• The checksum is part of building a rpm package. The “debsums” functionality is not actually required for all packages. In fact, until Ubuntu got their act together and started fixing a LOT of stuff, many debian packages, didn’t have their checksums in the db.

• Changes are atomic. They use the bdb transactions and rollback. Either something was installed (via CPIO) or it was not. Debian package manager uses flat text-files! Those flat text files live under /var/lib/dpkg [Figure 1 below]. There are about 3-4 of these files per package, and they often times corrupt, resulting in impossible to uninstall packages. This simply doesn’t exist with rpms.

• The Debian package format lacked multiarch support until Jan 31, 2012[2]. Up until this version, installing a 32 bit deb on a 64 bit operating system involved creating a full system 32 bit chroot (I shit you not!), with hundreds of megabytes of silliness. Rpm has had multiarch basically since the majority of Fedora compiled with 64 bit compilers (around early 2005).

Package creation o RPM packages have 1 command, rpmbuild, for creating a binary or source rpm. You have a single file ${package_name}.spec, and the source tarball. That is all you need for a rpm. If you want to build the package in a “clean room” chroot, you can use mock, which runs as non-root

o For creating a deb package, you have dpkg-source, dpkg-buildpackage, dch, etc. For debs: Do you use debhelper or do you use cdbs? What version of debhelper, what version of cdbs? Which is deprecated and which is the “preferred” way? You have to edit the control file, the rules file, the package list file, the changelog has to have the perfect format or all hell breaks loose, etc. If you don’t put the exact same info in the control file and the package.dsc, woe be unto you! Once you’ve got that all done, you have to create a “debian source package”. Don’t get me started on that stupidity, seriously, it is worse than this entire thread.

• Multiple utilities. There is rpm. Then there is dpkg, dselect, dpkg-query, dpkg-reconfigure, dpkg-deb. dpkg-<TAB><TAB> results in 35 results on my test Ubuntu box and almost 30 for deb<TAB><TAB> mostly debconf stuff and almost 70 for dh_* for debhelper grossness. There are a couple of helpers like rpmquery (shortcut for rpm -q), or rpmverify (which is a shortcut for rpm –V), but they are symlinks back to rpm for convenience. One utility, one man page, less ambiguity.

• Templating. A rpm spec file is simply a shell script with some substitutions. Debian packages are all built using some extremely customized autotools and autotools like macros, each with conflicting versions and competing implementations (try to figure out if you’re supposed to use cdbs[3] or debhelper[4]). Both debhelper and cdbs exist because it is so impossibly hard to build debian packages by hand without some serious pain. There are macros for rpm spec files, but not even remotely the same complexity or necessity.

• With dpkg, it is possible to get into states which are impossible to resolve with the cli utilities (even dpkg –-configure –a). This always results in having to manually edit the pre/post hacky script under /var/lib/dpkg and is serious voodoo black magic that only experts should ever do. The problem is that it isn’t uncommon. If the pre/post scripts do ever fail bad enough to where you can’t fully remove a package, you can do rpm –e –-noscripts. Debian packages have this “rc” state where they are partially installed / uninstalled, but not fully either. Then you have to purge them using dpkg –-purge, and that is assuming that you’ve successfully hacked up the scripts that read the plain text files under /var/lib/dpkg. The entire design is unbelievably fragile. They make it a bit less fragile by writing an ENORMOUS debian packaging policy[5] to try to get people to work around silly limitations in the software via policy. This is against one of the fundamental design choices of rpm, which is that package installs should be atomic. It is either installed, or not installed. There is no “I’m half installed” status for rpm packages.

Hopefully this is a reasonable technical defense of rpm’s superiority over dpkg. What blows my mind is that Ian Murdoch made Debian after Redhat Linux existed but before YellowDog wrote yum. He had time to study the internals of RPM and design something superior. Instead, he NIH and invented something that on most levels to this day is still technically inferior. The yum vs apt debian isn’t near as lopsided where old apt > old yum, but new yum is unbelievably > than new apt. That is for another day, and only if you’re interested.

[1] http://www.digitalprognosis.com/opensource/scripts/restorepe...

[2] https://lwn.net/Articles/485349/

[3] http://build-common.alioth.debian.org/cdbs-doc.html#id250485...

[4] https://joeyh.name/code/debhelper/

[5] https://www.debian.org/doc/debian-policy/

I don't have the knowledge to support or contest anything except your last sentence. yum's UI is the only thing I like more than apt-get. But yum doesn't have the concept of depends vs. recommends, so installing nginx, for example, REQUIRES pulling down geo-IP libraries that are entirely unnecessary.

Entirely agree with you. That is the singular area that dpkg used to beat rpm as a format. That isn't a feature of apt-get or yum, but of the underlying package format.

However, it is supported in a very new release of rpm:

http://www.rpm.org/wiki/Releases/4.12.0#Generalbugfixesanden...

""" New --recommends, --suggests, --supplements and --enhances query aliases for querying weak dependencies """

So it is coming along :)

RPM 4.12.0 brought in support for weak dependencies, which implements the tags Recommends, Suggests, Supplements and Enhances, which provides analog functionality from apt > http://rpm.org/wiki/Releases/4.12.0

This is in Fedora 21, so should be in RHEL 8.

Newest version of RPM 4.13 will implemented logical dependencies, e.g. Require: foo IF bar , Require: foo OR bar OR baz, which are far more usable than soft dependencies.

There are a number of errors and omissions here that I would like to try to clarify. These complaints seem to stem from a lack of understanding with dpkg and its workings, and some are rather... out of date. Just about the only thing I agree with is that I wish package checksums would be required by Policy!

> RPM starts off with the concept of pristine sources. It is vehemently rejected that a maintainer of a rpm package use a non-upstream tarball or change _anything_ without a patch that is in version control.

dpkg is exactly the same. A "source package" consists of a .dsc file, which is metadata, an .orig.tar.gz file, which is the pristine upstream source, and a .debian.tar.gz, which is extracted to a 'debian/' directory after the pristine archive is unpacked. Within the debian directory is the build system, including local patches, which live under debian/patches.

> Changes are atomic. They use the bdb transactions and rollback. Either something was installed (via CPIO) or it was not. Debian package manager uses flat text-files! Those flat text files live under /var/lib/dpkg [Figure 1 below]. There are about 3-4 of these files per package, and they often times corrupt, resulting in impossible to uninstall packages. This simply doesn’t exist with rpms.

I have seen the RPM database become corrupted several times over the years, including if I tried to install an RPM while low on disk space! I have never seen the same with the dpkg database. And dpkg is no less "atomic"; dpkg keeps track of a package's state, and if interrupted a new instance will clean up or resume whatever the old instance was doing.

> • The Debian package format lacked multiarch support until Jan 31, 2012[2]. Up until this version, installing a 32 bit deb on a 64 bit operating system involved creating a full system 32 bit chroot (I shit you not!), with hundreds of megabytes of silliness. Rpm has had multiarch basically since the majority of Fedora compiled with 64 bit compilers (around early 2005).

This is not a valid comparison. Multiarch in RPM means something completely different to what it means in dpkg. dpkg's multiarch system is much more powerful and complete.

> With dpkg, it is possible to get into states which are impossible to resolve with the cli utilities (even dpkg –-configure –a). This always results in having to manually edit the pre/post hacky script under /var/lib/dpkg and is serious voodoo black magic that only experts should ever do. The problem is that it isn’t uncommon. If the pre/post scripts do ever fail bad enough to where you can’t fully remove a package, you can do rpm –e –-noscripts.

This is 6 of one, half-a-dozen of the other. In the RPM world you end up without the cleanup from the script ever taking place, and so the package ends up leaving traces of whatever throughout your system.

I've never thought of editing the maintainer scripts under /var/lib/dpkg/info as "serious voodoo". They are simple shell scripts, written to dispatch off of argv in the machinism specified by Policy. Errors in good quality packages are rare and trivial to fix.

> Debian packages have this “rc” state where they are partially installed / uninstalled, but not fully either. Then you have to purge them using dpkg –-purge, and that is assuming that you’ve successfully hacked up the scripts that read the plain text files under /var/lib/dpkg.

This is a good feature of dpkg. It allows packages to be removed without their conffiles or other important data (e.g., your PostgreSQL databases) being removed along with them, unless the admin takes the manual action to purge the package.

> The entire design is unbelievably fragile. They make it a bit less fragile by writing an ENORMOUS debian packaging policy[5] to try to get people to work around silly limitations in the software via policy. This is against one of the fundamental design choices of rpm, which is that package installs should be atomic. It is either installed, or not installed. There is no “I’m half installed” status for rpm packages.

So then, what state is a package in if RPM is interrupted while unpacking files? dpkg puts the package into 'half-installed' state, from where you can run a new instance to continue the installation or remove the package.

As for CDBS or debhelper, the vast majority of packages in the archive use debhelper, and debhelper use is increasing over time. Developer documentation tells you to use debhelper. The rest of the complaints about the build process I really don't understand, but perhaps I have just been building packages for too long. I'll take 'dpkg -i foo.deb' over 'rpm -ivh', oh wait, I meant '-Uvh' any day. :p

> Just about the only thing I agree with is that I wish package checksums would be required by Policy!

Furthermore, < 350 binary packages out of ~50000 lack an md5sums file according to https://lintian.debian.org/tags/no-md5sums-control-file.html.

"Redhat never had issues such as this that Canonical did about licensing terms not being GPL friendly, which they have since fixed"

Sorry, I don't think that characterisation is entirely fair. First, the issues that were resolved was making wording clearer because the FSF was concerned that it was confusing. There was no fundamental changes to principles of the policy so there should be no implication that Canonical was somehow falling foul of GPL licensing. For clarity, as the post you linked to shows the FSF would like more done.

Second, the comparison to RedHat is completely unfair because the two companies follow fundamentally different strategies. Ubuntu has a single code-base, RedHat has a commercial and a free code-base. This means Ubuntu has a single IP policy to deal with both commercial and free usage. Whereas you sign a standard commercial subscription agreement, or a different agreement if you use RHEL in other ways. For clarity, I'm not besmirching either approach, I'm explaining the context of difference [0].

Thanks for adding that they were fixed.

I think you were generally trying to support this point:

"I suspect quite a few Redhat employees will take strong offense to your comments on them not caring about Free Software"

Which I fundamentally agree with you on. I've met lots of current and ex-RH people and they all care about Free Software.

This is more broadly true. Generally, people don't put their careers in areas they don't care about. I've almost never met anyone in any of the commercial Open Source companies that doesn't care about Free Software. There might be lots of opinions and arguments, but everyone is trying to do the right thing.

[0] As a disclosure I should point out that I was responsible for Canonical's IP policy until March of this year

There is, Ubuntu is like Debian with 5 years supported.

Or when I see Debian, I know that the system or person actually is serious, is likely to have a good operations and sysadmin mindset and will probably Just Work™.

No, that's Slackware.

I went through everything before I reached Ubuntu. I've been a Linux user since the early-mid 90s. This kind of stereotyping is inaccurate and harmful.