Hacker News Reader

Comment by jedberg | Hacker News Reader

I had to deal with fallout from this. At the time I worked at eBay, and were not allowed to use any OS except Windows or RedHat Linux. The reason we had to use RedHat? Because we had a paid license from them that included them absorbing all liability from IP claims regarding Linux.

But if anyone was around then, you'd probably remember that RedHat was one of the worse distros of linux at the time. So we were forced to use an inferior product because it came with IP indemnity. Thanks SCO!

View on HN

While I agree that the SCO FUD surrounding Linux was real at that time (we all felt it), Red Hat was one of the most respected, functional, and widely-implemented distributions for the entire time SCO was in the news headlines.

It was also a huge pain if my experiences (early-mid 2000s) with Fedora are any indicator. Red Hat and co. were great because, given enough effort, they could be tuned into unshakable systems for any purpose. And you paid for that with your time and energy (or a support contract with Red Hat itself) and occasional frustration at things that didn't work like any other distribution.

> But if anyone was around then, you'd probably remember that RedHat was one of the worse distros of linux at the time.

There's no better microcosm of the early Linux world than this. "Yeah, that giant market-driving lawsuit was bad, I was there too and suffered along with the rest of you. But that's not the important thing: let me tell you about how bad RPM is compared to DPKG! Did you know the underlying archive format was cpio? CPIO!"

That's not intended to be too much of a barb (I was a Debian nut too, FWIW), but really, all that stuff seems pretty silly in hindsight.

> Did you know the underlying archive format was cpio? CPIO!

And what's wrong with that? deb's underlying format is ar, which is basically the same class of simple archives that cpio is.

It solves the problem with two nice benefits:

1. You're not reinventing an archive format, the existing ones are perfectly adequate.

2. If your system is borked enough that you don't even have a working package manager, you can manually extract your package manager from a package file with nothing more than the cpio or ar programs.

> me tell you about how bad RPM is compared to DPKG!

I'm pretty sure the person you're replying to was taking about the distro as a whole, not the package file format

Yes, but the arguments of that era always seemed to revolve around minutiae like the package manager, or default filesysem, or choice of default desktop.

You're being dismissive of issues that were far bigger than you make them look.

>like the package manager

It wasn't about, for example, dpkg vs rpm, but apt-get versus.. nothing, because Red Hat had nothing to solve dependencies. Installing software on Red Hat was a truly hellish experience.

>default filesysem

I have never in my life seen people dismiss a distro because of a default filesystem. I question whether you've truly experienced the early days of Red Hat.

They shipped broken development snapshots of GCC that led to serious compatibility issues :

https://linux.slashdot.org/story/00/10/07/0027218/gccs-respo...

They employed one of the most abrasive and divisive personality to helm the development of the libc :

https://news.ycombinator.com/item?id=2380062

They were one of the most barebones distro in terms of tooling to configure and manage them. Debian was more fully featured on the terminal tooling front, while Mandrake and SUSE had a lot of GUI wizards for people more used to the Windows ways.

Even much later in the Fedora days, when Kernel developer Alanx Cox questions the sanity of releasing a completely broken distro (version 18 of Fedora) this is what they had to answer :

>So yeah: in case you didn't get the memo, F18 has a new installer and a new upgrade tool. They are both v1.0s. As in the case of all v1.0s, you may want to exercise some frickin' caution. If you want a Fedora release whose installer and upgrade tools were stabilized over a period of several years and 20+ releases, Fedora 17 is right in the torrent list

Uh, like.. okay? but maybe don't release it, until, you know, it's not garbage?

And then there's the whole thing with what Red Had did to take over the whole userland desktop stack with systemd, logind, dbus, pulseaudio, wayland.. all of which caused more issues than they ever solved. You still can't make software like autohotkey on wayland. All those pieces of software are heavily interconnected with each other making it less and less tenable to maintain a distro that doesn't package them.

There are many, many reasons to be soured with what Red Hat contributed to linux.

> You're being dismissive of issues that were far bigger than you make them look.

You whooshed on the humor above, so I should spell it out. The point is exactly the opposite. This focus on tribal distro warfare obscures the objectively much more important threat posed by this lawsuit to the entire ecosystem.

I mean, no, you're simply wrong. SCO was a far, far, far larger threat to the Linux world than the fact that Red Hat inexplicably dragged their feet shipping yum on their enterprise distro.

And it's important, as a matter of history, to remember that period and the players and the resulting and continuing effects on our culture. While on the flip side, no one cares (or rather: no one should care) about the lessons of apt vs. yum, all of which have been recapitulated a thousand times since. Let it go.

+1 to this. I remember installing software in 1998 on Red Hat and it was absolute hell. It was a hit or miss endeavor and when it failed, it failed badly. All those broken dependencies took ages to resolve.

Then I switched to Debian Potato and never turned back. It was such a clean and sane experience.

Unfortunately, I could not scape rpm hell at work as Fermi Linux, and then CERN Linux, were both based on Red Hat.

> It wasn't about, for example, dpkg vs rpm

No, it totally was. APT vs. nothing was a big deal, but RPM per se had a lot of mistakes. Debian also had massively better standard tooling around conffiles, diversions, alternatives, etc. They also had publicly-published packaging standards so even out-of-distro debs were often pretty good, while third-party RPMs were a garbage fire.

Nah, RH was very much it's own little world. Many of us were forced to use it because they offered a package (indemnification, support) that appealed to the bean counters and lawyers. For a few years they patched _everything_ that mattered (compilers, kernel, core libraries) to a massive extent in an attempt to differentiate and to give the appearence of adding value. I don't have fond memories.

There were several good versions of Red Hat (IIRC 7.3 was pretty good) and then RH9 had NPTL but unfortunately was incompatible with the rest of the world (very custom kernel, userspace, compiler). IIRC the egcs included in that led to all sorts of interesting outcomes.

7.3 was my first Linux distro, I distro hopped for a few years and settled on Gentoo in 2004...

Flash forward to a few weeks ago, I tried installing good ol' 7.3 again and it wouldn't even finish the installer...

I don't remember if there was a trick to it, or whether bit rot and architecture changes make it not installable anymore.

I'm confused. RH9 is the latest release, how is it custom? What are you referring to exactly?

RH 9 is different than RHEL 9.

Just like Windows 11 is different from Windows 98 (98 doesn't mean it's newer).

They went through a rebranding from RedHat Linux to RedHat Enterprise Linux over a decade ago but after or during the SCO nonsense. Early versions of RedHat's Linux were just called RedHat Linux because at the time it was just like any other Linux distro or starting from those early hobbyist roots. RHEL was meant to be stable and have commercial add ons for big companies.

https://en.wikipedia.org/wiki/Red_Hat_Linux

https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux (at the top it notes, not to be confused with Red Hat Linux)

What could be thought of as Red Hat Linux today is the Fedora distro (which it was spun off into), which is upstream, more frequently released or up to date (and thus less stable), and targeted at the hobbyist or enthusiast user, which I'm one of the weird people that still uses it. Once something useful goes through its paces on Fedora, it'll eventually get integrated into RHEL where reliability is of higher value than the latest features.

I started on RH 5.2 myself, in '99, and literally got it on disc for $40 at BestBuy. I also lived near RedHat's offices (in Durham, NC) at the time and got to visit them once with coworker friends. Exciting time, I was very young, and wish I knew enough to buy stock during its IPO at the time.

No, the latest release is rhel9, not rh9.

At the time it was just “red hat linux” while today it is “red hat enterprise linux”.

The split (and the counter reset) was made to mark the split between a regular distro (fedora core, at the time) and an enterprise-oriented distro (red hat enterprise linux).

Red Hat 9- https://en.wikipedia.org/wiki/Red_Hat_Linux was in 2003. I may be misremembering some details since I thought RH9 brought in egcs but it was 7.3 that did that.

There was a distro called Red Hat, which more-or-less mutated into Fedora, which is and has always been a different distro from Red Hat Enterprise Linux.

Rh 4.2 ran half the planet, where are you getting this from?

From my experience as a Linux sysadmin? RedHat ran half the planet because they had the best sales force and legal team (as exhibited by my example), but they did not have the best technology.

You are not wrong...

It looks like RHEL zealots never die from the other people’s comments. RHEL had plenty of problems just like all the other OSes and distros.

I think you're romanticizing it. Until RHEL and LTS releases for Debian/Ubuntu, most distros you never knew if running and update was going to break something because there simply wasn't effective quality control testing in the hobbyist distros. Best you could do was run a version behind, but that hurt if you needed security updates.

There were plenty of people and small highly knowledgeable shops and academics that thought 6 hours fixing a bug after custom compiling a patch was fine and normal (and a RHEL subscription at least meant RedHat would have a team doing that part for you if absolutely necessary), but its not the way companies operated. RHEL at least meant whatever release was stable and an actual QA team put patches through their paces on various hardware and configurations (especially those enterprise high end server configs with special SCSI/RAID controllers, high end network cards, and other chipset other distros simply didn't have the means to test on). The QA/support team wasn't bug reports and guys on usenet going "it works for me, you should have gotten the exact same hardware I have, or be willing to go through the code and figure it out and patch it, and submit it to the source, like a good user should". Or tell you go back to Micro$oft if you want support for your storage controller that the kernel module for worked fine in the last version. Those were the zealots, the rest were sys admins with too much other things on their hands to do than deal with Slackware or whatever the hot distro was on distrowatch.

My Redhat experience always seemed to devolve into "this package that I want has a dependency that isn't listed yet..." (cue 2 hours of recursively and manually tracking down dependencies on the early web).

But I was a lot younger and didn't know a lot of what I do now, so was probably doing everything RPM wrong.

I had the dependency spiral issue on every distro (I played with quite a few), compiling with something like Gentoo made it worse. RPM Forge existed, I think the Linux experience in general was bad back then, and RedHat actually was one of the least problematic. Until Ubuntu, it was the easiest and most approachable to use for novices.

Using redhat before yum, meant visiting rpmfind.net and manually collecting what you needed.

In some ways, RHEL is still like that, because popular packages are usually a major version or two behind if they're even there at all. You have to hunt down an EPEL that has whatever you need.

Yeah, I don't miss the old days. Like another poster though, I remember apt-get being decent while pre-yum RedHat was still pretty bad.

But I also realize my perspective was that of a hobbyist, not an enterprise sysadmin who was probably upgrading to well-known versions through known paths.

I worked with sysadmins who used rpm based distros back then and their experience was basically mine: hunting down the right rpms that both satisfied the constraints and actually worked.

I think you are exaggerating.

Computers of all stripes are more reliable now. In the late nineties I ran apache + mod-perl built from source.

We had a lot of problems, but I cannot recall ever having a problem with those two.

We tested every update, of course, but it was not a huge burden.

If you were running Apache fine, I think Linux in the late 90s was the work horse of web servers. That might have been the one thing that just worked. I was using it as a workstation and for everything else. Try getting your window manager to work with your Xserver and graphics card. Upgrade a package that needs a newer packlage, that needs another package that no one has built yet, so now you need to custom compile the library, but if it replaces the existing library it breaks something else that relies on the older library.

That's inbetween figuring out how to get things to compile and the dependencies of dependencies, etc.

I got Linux to work, but it was also a love hate relationship, when I got it working, it worked and worked for months, but I had almost a PTSD reaction when it was time to upgrade anything, I knew what was coming and I was afraid.

You bring back memories. In my early Linux days, every attempt to upgrade an RPM based distribution lead to me needing to wipe the system.

The first time I did a major version upgrade on Ubuntu, I was shocked it worked.

Wrong. Two reasons actually:

1. RHEL was the first distro directed at enterprise deployment (meaning, strong preference of rock solid stability and predictability over constant churn). Which made it the only distro Dells and HPs of the world recognized and agreed to support.

2. RHEL was created on legacy of RedHat Linux, which was the best distro for non-hobbyist environments (from reproducible deployments to the breadth of packages available) - since 3.0.3 onwards. RedHat JUST. WORKED.

My experience disagrees strongly.

1. Debian always had more stability and predictability than Red Hat in practice. Too much so. The Dells and HPs of the world didn't recognize it because it was not a company.

2. My impression was again that Debian was technically better than RedHat in every way I might care about. We happily installed it at $work, and the experienced Unix sysadmins I knew could use RedHat but didn't like it so much.

> Dells and HPs of the world didn't recognize it [Debian] because it was not a company.

IIRC Debian maintainers formed companies for that reason.

I was a Debian user because it suited me. It was obvious (to me at the time) that suits were going to choose RedHat, and we ran it a bit to have experience.

How wrong I was. Ubuntu made Debian every bit as much as a "choice of the suits" as Redhat.

I don't think "Wrong" is a very interesting, helpful, or productive response to someone's lived experience.

This was pre DNF/YUM days. RedHat could be a pain installing stuff from RPM’s that had crazy dependencies back in the day.

First time I installed Linux, I was trying to Gaim on it.

I managed to compile Gaim before I managed to install it from RPM packages.

Using Debian was a moment of pure epiphany.

Good times.

I remember using Mandrake (later Mandriva) - it also used rpm. Dependency hell was an issue there too; it probably had to do with state of rpm more than the distro itself.

Dependency hell was a thing for sure. I had at the time made a script to easily install rpm's at with a shell script that was searching the mirrors. It didn't handle dependencies but would show what was missing. But so many years later now looking back, a big issue was really just a lack of understanding of rpms with all the other os's with rpms, different arcs and versions. yum was a welcome changed especially since apt had for so long solved dependency issues.

up2date was before then

My main experience with RedHat has been needing to bend over backwards to support its wildly outdated library versions. Because RedHat “supports” operating systems for about a decade, there’s always an argument that a library should be written to support whatever toolchain is provided by the oldest supported RHEL version.

RHEL’s “support” should be seen as “your software will continue to run unmodified when on this system”, but is frequently interpreted as “this is a sane platform for current development”.

For a while, Python binary wheels (packages) for Linux had a standard that was defined, essentially, in terms of RHEL (or rather, CentOS) as a baseline platform for this exact reason.

https://peps.python.org/pep-0513/

https://peps.python.org/pep-0571/

https://peps.python.org/pep-0599/

Current development was better done on Fedora, the upstream of RHEL, I think they were pretty clear that's what it was for. RHEL was for when you needed everything to work no matter what. Fedora was for new development and latest and greatest.

The problem with needing “everything to work no matter what” is that it just doesn’t exist. The type of stability that RHEL provides is conditional on never using third-party libraries. RHEL backpoets security fixes to packages that it provides, but you’re on your own for anything else. For example, RHEL 7 was released in 2014, has production support for another year, and extended support until 2026. As libraries are requiring C++17 support today, maintaining compatibility with stock RHEL 7 requires running older versions of a library.

That level of backwards compatiblity with legacy versions exists on other platforms (Windows, z/OS, AIX), especially those implemented in enterprise environments. If I have a Windows app written in Win32 API, it'll run on Windows 11 on 64-bit architecture. That's what companies want. That's how Linux became viable for those customers. Even Edge has an IE mode because that's what enterprise environments need, support for older libraries and APIs. I won't deny RHEL was higher technical debt for sure, but that was the trade off, because that's what enterprises prioritized, support, stability, and knowledge that an update during their monthly patch cycle and change window for rebooting the server won't break the business applications they rely on for fundamental operations, at least not on the same major release version.

RHEL was never supposed to be the latest, I think even a every new major release they'd be on a kernel and library version that was 2 years behind but had been through its paces. IOW, it was a feature not a bug, and its what companies were paying for (their customers were boring legacy Fortune 500s not startups). As a business model, it was solid, and the most successful in the commercial market by far because it catered to their customers needs, even if they're not our own, even Ubuntu Server took pages from their book.

They're talking about stuff that predates RHEL by half a decade.

True, more adding it as agreement on RedHat being consistently and intentionally behind the curve.

Windows ran half the planet, where are you getting this from?

These statements are not mutually exclusive. Math checks out :)

Only if MS and RH were the only thing that existed 20 years ago, which we know is false. Sun Microsystems, for one. Debian for another.

And the thing is I was being conservative about MS running half the planet. They had their grubby hooks in everything.

For sure. My comment was in jest.

> Windows ran half the planet

That half of the planet did not run....

That was Debian and FreeBSD not RH.

> Because we had a paid license from them that included them absorbing all liability from IP claims regarding Linux.

Not a trivial thing by the way, if you think about it.

Ahh yes. We also kept our hands and feet inside the roller coaster with SLES and RHEL.

The ever-looming “threat” might be laughable in hindsight had it not stifled innovation at such an early stage of growth.

IT Directors everywhere should atone.

What were the best distros then?

When I was a CS major in 1999-2003, the big names I remember people using were RedHat, SUSE, Debian, and Mandrake was gaining market share for desktop use. The installfests I went to were all RedHat. There seem to be a lot of complaints here about RedHat after that era, but my memory is that RedHat was the main linux distro circa 2003.

Mandrake! What a blast from the past. I also remember Gentoo and Slackware from around the same time, though apparently the latter predates the former by a decade or so.

I also remember Gentoo, mostly from here:

https://www.shlomifish.org/humour/by-others/funroll-loops/Ge...

https://web.archive.org/web/20170602132852/https://fun.irq.d...

I'm equal parts gobsmacked that funroll-loops is still around, and that it's thanks to Shlomi Fish who is still writing his creepy-ass fanfic.

Slackware in the 90s was amazing. Super stable. I think I continued to run Slackware until around 2002 to 2004 (can’t say for sure but it was to run an early(ish) version of Ableton for laptop DJing - as I was creating a concept set that just wouldn’t have been possible with vinyl alone), and wanted a distro that was a little lower maintenance given the advances to Linux at that point.

I’ll always have a soft spot for Slackware even if I’d never dream of running it any longer.

Aaaaah, so that's why <company I interned at a few years ago> used RHEL!

Reply on news.ycombinator.com