Hacker News Reader

Comment by binarycrusader | Hacker News Reader

Unfortunately, that's really the result of the API/ABI choices that the Linux kernel developers have made. There are no stable interfaces for kernel modules, and they frequently break existing interfaces.

This means that any third-party driver that is not included in the mainline kernel source is effectively broken either via API or ABI almost every kernel update.

So it's not specifically a proprietary driver issue since even third-party open source drivers that are not in the mainline kernel break too.

Yes, nVidia could choose to open up their driver and contribute it in a way that it was included in the mainline kernel and therefore breakages would happen far less often. But ultimately, the real problem is the API/ABI instability on Linux.

With the above said, as others have mentioned, it shouldn't generally break every update if you have dkms, etc. setup properly.

View on HN

ABI stability is not a feature, it's a bug. It means you can't fix the kernel and every driver in it, and you have to keep legacy interfaces around. This isn't just about keeping old function entry points around; the kernel regularly redesigns and rearchitects core subsystems in ways that cannot support the previous function entry points, and that's completely OK because the developers making those changes go fix all the drivers.

So no, the kernel will never have a stale^Wstable ABI for drivers (as opposed to userspace). Out of tree modules are a bug; pick hardware that doesn't need them. Graphics are one of the last areas where some of the widely used hardware still doesn't have FOSS drivers; just about everything else works with in-tree drivers.

And the stability issues aren't just with keeping the drivers up to date. Users who run the nVidia proprietary driver regularly report crashes and stability issues.

  ABI stability is not a feature, it's a bug. It means you
  can't fix the kernel and every driver in it, and you have
  to keep legacy interfaces around.

Sorry, but I strongly disagree and that's a factually, anecdotally, and objectively incorrect statement. OS X, Windows, AIX, and Solaris all manage to provide stable interfaces for drivers while still evolving the underlying implementation significantly. So clearly, it can be done.

A properly designed API/ABI allows evolution of the implementation while retaining compatibility through interface.

  Out of tree modules are a bug; pick hardware that
  doesn't need them.

The idea that the Linux kernel should or can contain all support for all hardware ever is an ideological pursuit at best.

  And the stability issues aren't just with keeping the
  drivers up to date. Users who run the nVidia proprietary
  driver regularly report crashes and stability issues.

And people still report stability and crash issues with mainline kernel drivers too, just perhaps less frequently, so that's really an orthogonal concern. Quality of implementation is not an argument I would want to use against good architectural software practices such as having a stable API/ABI. I'm also quite familiar with the argument that "if the source is open, people can fix it and make it better!" -- I think the recent OpenSSL problems have proven that just because the source is open doesn't mean anyone is fixing problems there.

Regardless, the real losers here are the users; I don't frankly care about the proprietary vs. not arguments and holding up some ideological standard as the ultimate goal seems like it's missing the point -- computers are just tools, and we should spending more time on improving the usability of them for users not fighting ideological wars.

> Sorry, but I strongly disagree

Fine.

> and that's a factually, anecdotally, and objectively incorrect statement.

No, your anecdotes do not constitute facts or objectivity. More importantly, I'm not the person you would have to convince, and the people who would need to be convinced consider this argument over and done with long ago.

> OS X, Windows, AIX, and Solaris all manage to provide stable interfaces for drivers while still evolving the underlying implementation significantly. So clearly, it can be done.

By accumulating a massive amount of cruft that they can never remove. Yes, it's possible; it's also possible to run the old kernel in an emulation layer to run older drivers, but that doesn't make it a good idea. (Though that'd probably still be a better idea than attempting to support the old in-kernel APIs in a newer kernel.)

And incidentally, if you need kernel ABI stability, you can also go run RHEL, which backports changes from newer kernels while maintaining in-kernel ABI stability for out-of-tree drivers, within a given major release. So for the support lifetime of that RHEL release, you can count on ABI stability.

> The idea that the Linux kernel should or can contain all support for all hardware ever is an ideological pursuit at best.

And, in practice, a shockingly successful one. There's very little mainstream hardware left that the kernel doesn't have in-tree support for. (No, that's not an invitation to enumerate the remaining exceptions.) And even for a fair bit of nVidia hardware there's the reverse-engineered nouveau driver, which is good enough for desktop environments and light gaming on the hardware it supports.

However, for a production-quality Linux system, it makes more sense to select hardware with in-tree drivers.

> And people still report stability and crash issues with mainline kernel drivers too, just perhaps less frequently, so that's really an orthogonal concern.

I never suggested it was related; I'm arguing that there are good reasons to not use the nVidia drivers other than that they're out-of-tree and proprietary, namely that they're notoriously crashy.

  No, your anecdotes do not constitute facts or
  objectivity. More importantly, I'm not the person you
  would have to convince, and the people who would need
  to be convinced consider this argument over and done
  with long ago.

Then why are you still arguing it? Why are your anecdotes somehow superior to mine (if you want to call an entire mainstream market "anecdotal" evidence)?

  By accumulating a massive amount of cruft that they can never remove.
  Yes, it's possible; it's also possible to run the old kernel in an
  emulation layer to run older drivers, but that doesn't make it a good
  idea. (Though that'd probably still be a better idea than attempting to
  support the old in-kernel APIs in a newer kernel.)

No, and no. As a kernel developer (my day job) I can assure you that is not the case. And note I never said that you have to maintain ABI/API compatibility forever. Even operating systems with stable API/ABIs do change them from major release to major release.

The point is, the Linux kernel could successfully adopt a stable ABI/API without accumulating all of the "massive cruft" you claim is impossible to avoid and make lives better for their users. At the very least, providing an ABI/API they guarantee to be stable for some period of time instead of breaking things almost every point release.

  And incidentally, if you need kernel ABI stability, you can also go run
  RHEL, which backports changes from newer kernels while maintaining in
  kernel ABI stability for out-of-tree drivers, within a given major
  release. So for the support lifetime of that RHEL release, you can
  count on ABI stability.

So in other words, there's a genuine need for a stable API/ABI and RedHat supports it, but you still think it's stupid and useless to have and there's clearly no need for Linux to do it, except RedHat has proven there is a market for it. Hmmm....

  And, in practice, a shockingly successful one. There's very little
  mainstream hardware left that the kernel doesn't have in-tree support
  for. (No, that's not an invitation to enumerate the remaining
  exceptions.) And even for a fair bit of nVidia hardware there's the
  reverse-engineered nouveau driver, which is good enough for desktop
  environments and light gaming on the hardware it supports

"Pay no attention to the exceptions; it's a complete success!" The exceptions are proof that it's not a complete success and yes, quite frankly, some hardware that is not widespread enough or is special to a given organisation doesn't belong in the Linux mainstream kernel. What would be the point of "accumulating all that cruft" after all...

  However, for a production-quality Linux system, it makes more sense to
  select hardware with in-tree drivers.

Right, because all of those Linux systems at DreamWorks and other places with nVidia cards in them aren't "production-quality"; oh wait...

And how many of those various crashes and other problems reported with nVidia systems are due to the rapid changes in the Linux kernel instead of the driver itself? You could argue it's hard to say without source to the nVidia driver, but that seems like a tautology at best.

It's fine that we disagree, but your assertions thus far are easily contradicted by the entire mainstream market (nevermind your own RedHat example) so sound kind of silly.

> Then why are you still arguing it? Why are your anecdotes somehow superior to mine (if you want to call an entire mainstream market "anecdotal" evidence)?

I'm giving the stated reasons for why Linux intentionally has no stable in-kernel ABI, and what some of the downsides of having one are. That's not an anecdote, that's a description of the stated design philosophy (see also https://www.kernel.org/doc/Documentation/stable_api_nonsense... ). You gave exclusively upsides to having a stable ABI, as though it was an obvious design failing of the Linux kernel, rather than a tradeoff.

To be clear: I absolutely agree that it's possible to provide driver interfaces that don't change; as you said, systems like Windows provide evidence of that. But that doesn't come without a cost.

But I'd be happy to be proven wrong; just point to the source code of one of the kernels you mentioned where they implement a stable kernel ABI without any accumulated cruft...oh, wait. ;)

> And note I never said that you have to maintain ABI/API compatibility forever. Even operating systems with stable API/ABIs do change them from major release to major release.

You gave no indication that you ever considered it acceptable to break ABI. I'm certainly in agreement that ABI shouldn't be broken gratuitously for its own sake; however, it shouldn't be necessary when making a change to the kernel to think about whether out-of-tree drivers (that nobody can see the code of) depend on the behavior of a particular function call, or the layout of a data structure, or the context a given function can be called in.

> So in other words, there's a genuine need for a stable API/ABI and RedHat supports it

For RedHat paying customers who intentionally want to run the same software for 5+ years at a time and avoid any changes, sure. It makes no sense for the upstream Linux kernel to give those same guarantees while doing active development. 5 years ago was roughly 2.6.36. That's 5 years and ~24 kernel releases where not a single field can be added or removed from a data structure, no argument can be added or removed for an existing function, and semantics like "what lock do you need to hold to call this function" cannot change. (There's a lot more to ABI than just function calls and data structures, particularly when you're talking about a C ABI between privileged code running in the same address space.) Maintaining compatibility like that would completely hamstring day-to-day development.

If you have that kind of compatibility requirement for a kernel, and that kernel remains under active development (rather than being declared done and untouchable, which is another option used for some embedded systems or microkernels), you'd need to heavily rework your architecture and development processes to insulate the actively-developed kernel from the drivers, generally by introducing something like a HAL, so that the kernel can change as long as it exposes a compatible HAL. Then you get into issues like HAL versioning, including potentially supporting multiple versions at once, interactions between drivers and interfaces provided by one driver to another, and so on. All for the sake of offering better support for drivers that refuse to submit their code to the upstream Linux kernel.

So yeah, it's possible to build a kernel that provides interfaces that drivers can rely on across many versions. Not without a cost, though.

  So yeah, it's possible to build a kernel that provides
  interfaces that drivers can rely on across many versions.
  Not without a cost, though.

Software engineering is all about tradeoffs and making the right ones, I feel the Linux kernel community has made the wrong one by forcing those costs onto users instead of helping absorb some of that themselves.

I never said it was free or that there wasn't a cost; I just said that usability is important and have implied the Linux kernel developers should care more than they do now.

That was why I brought up RHEL: they're not pushing it on users, they're pushing it on distros whose users care about binary driver compatibility.

It's sort of silly to imply that only RedHat users care or that only RedHat users should benefit. All users need and benefit from stability.

Only people running long-term-stable distributions care, and those distributions provide the level of ABI stability their users want. I'm simply saying that it makes more sense for downstream distributions to handle rather than the upstream Linux kernel.

On the flip side, users running up-to-the-minute distributions don't necessarily care, and in fact care more about some of the properties that a stable upstream kernel ABI would trade away.

More importantly, if a stable ABI does not exist upstream, a downstream distro can add it. But if a stable ABI exists upstream, a downstream distro can't get the traded-away benefits back.

So only some distributions get a stable ABI but not others, leaving users of distributions with less support with a less usable system.

I sincerely question the supposed benefits of this tradeoff.

I have yet to see any articulated beyond hypothetical improvements in ease of implementation and zero benefit to the average user.

Distributions whose users want an unchanging ABI get one. Distributions whose users don't care (e.g. those where either the users don't care about proprietary/out-of-tree drivers or the users use the packaged versions of those drivers in the distribution that get rebuilt when the kernel changes) don't bother.

Your arguments assume that everyone obviously wants an unchanging driver ABI. I'd love to see statistics for how many users actually build and run out-of-tree drivers themselves, but I'd bet on it being a small fraction, most of which is the nVidia or ATI proprietary graphics drivers. And again, personally, I don't consider it a feature to cater to proprietary driver vendors.

Beyond that, I've seen several reports regarding some of what binary driver vendors do that goes well beyond the exported driver interface (e.g. searching for core kernel data structures in memory and grubbing around in them), so it's not necessarily obvious that an unchanging driver ABI would actually stop binary drivers from breaking.

In any case, I'm not interested in arguing further about the development practices I've already mentioned and pointed to that depend on the ability to evolve an interface along with all of its users.

There are significant benefits that Linux has derived from its lack of a stable driver ABI.

For one example, adopting the AMD64 architecture proceeded significantly faster on Linux systems because nobody had to wait for binary drivers to become available, thus enabling end users to use their shiny new hardware to the fullest extent.

For another, do you believe that the effort to add real-time capabilities to Linux would have any chance at all at succeeding while being constrained by stable internal ABIs?

Basically only GPU drivers are sufficiently complex that the vendors are actually concerned about protecting their IP investment there and release only binary drivers; though AMD has even figured out recently that they can open source the kernel part of their GPU driver since all the interesting stuff is happening in user-space.

Are driver ABIs of other kernels actually stable across major versions, in all cases? Another comment of yours says this is not the case.

One major difference here is that there's a new release of Linux with substantial new features every 3 months, whereas new releases of NT come out every 3-6 years, and the gap is even larger for Solaris.

Another major difference is that the upstream Linux kernel doesn't have paying customers; products like RHEL and SLES do, and maybe that is the level at which things like driver ABI guarantees should be provided, if there is actual demand for it.

Reply on news.ycombinator.com