Google discloses CentOS Linux kernel vulnerabilities citing failure to fix | Hacker News Reader

Google discloses CentOS Linux kernel vulnerabilities citing failure to fix

neowin.net

159 pointsjaboutboul76 comments

It's a not so well guarded secret that the claims by many linux distributions that they support old systems with security fixes for many years are not really true. The number of security flaws constantly found and fixed is just too large to make this even remotely feasibile. This is just another instance of this.

OTOH, a lot of customers probably aren't too worried about low severity, local attack vector bugs.

Can confirm this is a very real problem with a long history. Quoting from a sort of Arch-specific security hardening page[1]:

> A common misconception about the Linux kernel is that it's secure, or that users can go a long time without worrying about security updates. Neither of these are even remotely true. New versions of Linux are released almost every week, often containing important security fixes among the other changes. These releases typically don't make explicit mention of which commits have security implications. As a result, many "stable" or "LTS" distributions don't know which commits should be backported to their old kernels, or even that something needs backporting at all. If the problem has a CVE assigned to it, maybe your distribution will pick it up. Maybe not. Even if a CVE exists, at least in the case of Ubuntu and Debian especially, users are often left with kernels full of known holes for months at a time. Red Hat and similar "enterprise" distros have the same problem and have been called out publicly about it on more than one occasion. Moreover, the Linux kernel security team doesn't request CVEs for any vulnerabilities, partly because there are just too many to track. Downstream's culture of trying to cherrypick security fixes in the name of stability does not work. Arch doesn't play the backporting game, instead opting to provide the newest stable kernels shortly after their upstream release.

[1]: https://vez.mrsk.me/linux-hardening.html (lots of linked sources in the actual paragraph there)

This is one reason why I don't mainline a Linux desktop anymore. It's just too much sysadmin work to stay on top of it, in addition to all of the desktop issues.

When even the creator of the Linux kernel has a very lax approach to security, and so the culture of Linux is the same way - I consider it an issue.

Is there any real reason you don’t auto update everything, especially the kernel as a desktop user?

This is one of the nice things about arch I guess. Super simple to run mainline.

Experience has taught me not to be the first to beta-test a new package update, and don't blindly update. I used to run Arch and have run into scenarios multiple times where I've been unable to boot into a desktop environment because of an update. They were recoverable issues, but still cost me time and motivation for that day.

That, and updates are just a constant stream - there is always a new package to update every single day.

And as a separate issue, but the quality of software has taken a sharp decline lately as we all start switching to flatpaks and snaps (a correlation, not causative). More and more software is Electron-based and the quality of the non-Electron based software has more or less tanked.

Not many hobby programmers are making desktop software anymore.

You could try using Nix OS stable with "boot.kernelPackages = pkgs.linuxPackages_latest;". That way, you will get the latest kernel version and if there is an issue with you being unable to boot into a desktop environment, you can very easily roll back the update by rebooting and selecting an older generation in the boot manager's menu.

I second that. On few occasions kernel update in Fedora on my Dell XPS from 2016 cannot boot. The last time it was an upstream regression that was not fixed for months. In the mean time a bad WiFi bug was discovered. I could in theory compiled my own kernel with a fix but without the boot regression, but I rather decided to use Debian. At least they do backport fixes

Nothing prevents you from running the latest kernel. The kernel build scripts can even generate .deb and .rpm directly so it coexists nicely with the package manager, though even that isn't strictly necessary.

Other than the constant restarts necessary. It’s not that but it does get old fast.

kpatch has been around for like a decade from Red Hat, I think suse has something similar, ubuntu has a Livepatch service.

Who cares about a few reboots? You can reboot a workstation overnight and if you have a single point of failure in your production system the reboot ain't the problem.

> You can reboot a workstation overnight

That depends entirely on how well your state serializes to disk and restores. If nothing else, I occasionally make the mistake of getting a setup going that would just be a pain to recreate in terms of getting everything open again (too many windows open in a specific arrangement).

Nothing prevents you from running the latest kernel, but the project is introducing new bugs with security implications all the time even as it is eradicating old bugs, which is why one guide recommends an LTS kernel as the better of two bad options (the other option's being a stable kernel):

https://madaidans-insecurities.github.io/guides/linux-harden...

Newer bugs are get fixed faster and are less likely to be exploited compared to older bugs.

Ubuntu LTS has a package called linux-hwe that backports the latest Ubuntu kernel to the last LTS release. I find it's a good compromise between maintaining a mainline kernel and using an ancient one. And since the hwe kernel is provided by Ubuntu and used in its release, it's been tested by thousands of users and updating it is a breeze.

My only complaint is that I wished it went back 2 LTS releases. For example, the current project I'm working on settled on 20.04 and there are no more new hwe kernels being released for it despite the rest of the userland receiving updates.

Nothing other than breaking my graphics drivers. Who needs a monitor?

A few years ago we couldn't upgrade the kernel because XFS had a regression. Our choices were: revert the XFS changes in the newer kernel; backport security fixes to the version we were already using.

We went with the latter because our minimal attack surface made it unlikely that we'd need to backport many fixes, but it was always looming.

Edit: I have been corrected on the support time frame. Also had the bad luck of checking the CentOS wikipedia page to validate my memories - one of the Tables makes it look as if both CentOS stream 8 and 9 are already out of support.

As far as I can tell IBM officially nuked CentOS long term support in the same step it killed Red Hat compatibility with 7 being the last supported version.

The replacement for CentOS seems to be Rockit Linux, which continues with tracking Red Hat packages like CentOS did in the past.

I always looked upon Scientific Linux as the CentOS replacement.

https://scientificlinux.org/

Scientific Linux was killed in 2019 although security releases for SL 6 and 7 will probably continue as long as Centos 6 and 7 get them. I'd go with AlmaLinux or Rocky Linux if you want to something that's a free copy of RHEL 8 or 9

CERN chose to move forward with AlmaLinux: https://linux.web.cern.ch/which/

That's mostly replaced by Alma with the CERN repository: https://linux.web.cern.ch/which/

RockyLinux is by the guy who did CentOS, and seeks to follow it's principles.

Scientific Linux is no longer a thing, essentially; from wikipedia:

> In April 2019, it was announced that feature development for Scientific Linux would be discontinued, but that maintenance will continue to be provided for the 6.x and 7.x releases through the end of their life cycles. Fermilab and CERN will utilize CentOS Stream[4] and AlmaLinux[5] for their deployment of 8.x release instead.

> RockyLinux is by the guy who did CentOS, and seeks to follow it's principles.

Not quite. Back in 2003, when folks were first coming up with the idea of creating a distro that matched RHEL as closely as possible, Greg Kurtzer (the Rocky founder) explicitly stated that he would be interested in helping host it, but that he was "totally not interested" in leading it.

https://web.archive.org/web/20040909231336/http://www.caosit...

It wasn't until much later (~2019) that he started trying to retcon history by claiming he was the original CentOS founder.

That is not true.

CentOS Stream is just as "LTS" as Debian, Ubuntu, or OpenSUSE - supported for 5 years. This is admittedly not as generous as the 10 years CentOS provided previously (and what Alma / Rocky linux have promised to provide), but it's definitely still "long term" by the same definition everyone else uses.

And while CentOS Stream {8,9} does trend slightly ahead of RHEL {8,9} respectively in terms of bugfixes and features, it still maintains equivalent ABI with RHEL {8,9} and is compatible, though not precisely "bug-for-bug" compatible, which even CentOS / Alma / Rocky have never quite accomplished.

disclosure: I work for Red Hat

That is probably what I get for checking my assumptions against wikipedia. One of the tables on the CentOS page makes it looks as if only CentOS 7 is still supported. On a closer check the table may just be incomplete. https://en.wikipedia.org/wiki/CentOS

Oof, that table is very wrong. Even the release date for 8-stream is wrong. Thanks for pointing this out.

Rockit?

Oracle Linux 7 is guaranteed binary compatible with RHEL/CentOS 7, although there is only (a little over) a year of support left on the whole platform.

Scientific Linux was also on v7, but I thought there was some kind of ongoing apoptosis with this repackaged distro.

Neither Alma nor Rocky implemented v7. I thought that CentOS left that version untouched.

I wonder if the Oracle UEK has addressed these problems. If so, it would be a reason to run Oracle's kernel on CentOS, supported or not.

For those poeple who can tolerate neither these bugs nor Oracle's kernel, another option is ElRepo's "Kernel of Last Resort."

"We stress that we consider such kernels as a last resort..."

http://elrepo.org/tiki/kernel-ml

UEK can run into the same issue. It is just a picked upstream kernel with a bunch of stuff that Oracle cares about dropped on top of it, and managed by a much smaller team.

It can run into the same issue, but they are all v5 kernels.

Rhel7 is running a v3 kernel; the UEK will have a lot less to do.

I will admit that they are with a few point releases on rhel9.

How many people maintain elrepo-ml?

There are many parts of Google that I won't rise to defend, but to preempt the inevitable: Project Zero does disclose Google issues on the same basis as anyone else's. https://bugs.chromium.org/p/project-zero/issues/list?q=label...

It wouldn't matter if they didn't. Research and disclosure of vulnerabilities is an absolute good; they don't need to earn the right to publish bugs (it's closer to the other way around).

This mentality is becoming passé. And frankly this article/post is a good example of the reason. It was long thought that you could name & shame people into good security, but it turns out they just purchase cyber insurance and retreat deeper into the shadows, which increases exposure to the end user.

But the new reality is that malicious uptake on publicized POCs is so great, and the negligent refusal to patch is so great that a widespread culture of reporting has actually increased harm to the public. That is the reality, and until there are laws that hold companies liable for the software they choose it will remain the case.

I came up in an era of CERT's anti-disclosure regime, with no auto-update anywhere, and there is simply no comparison. No, to all of what you said; I don't think I could disagree with anything more strongly.

You don't know what issues they withheld.

From the project zero posting [1] it would seem like most of these would be relatively low scoring CVEs and hence maybe less of a reason to prioritize a fix.

[1] https://bugs.chromium.org/p/project-zero/issues/detail?id=24...

The CVEs are for CentOS Stream 9, not the RHEL derived CentOS 7 or Rocky/Alma 8/9. If the RHEL derived distributions were patched but not Stream it would seem that IBM-RedHat's promises about CentOS Stream being just as good are now provably false.

I'm not sure if such a promise ever existed. After all, before CentOS Stream, all security fixes went out with RHEL first and were imported into CentOS (non-Stream) only afterwards. And under the Stream development process, the fixes have to be merged into Stream eventually for the next minor RHEL release, otherwise they would be missing from that, too.

The CVE tracking pages are here:

https://access.redhat.com/security/cve/CVE-2023-0590

https://access.redhat.com/security/cve/CVE-2023-1249

https://access.redhat.com/security/cve/CVE-2023-1252

Based on these data, it seems that the commercially supported versions are still unfixed (if they were impacted in the first place).

Bugzilla records show that the corresponding security tracking bugs were filed publicly from the start:

https://bugzilla.redhat.com/show_bug.cgi?id=2165741 (CVE-2023-0590)

https://bugzilla.redhat.com/show_bug.cgi?id=2169719 (CVE-2023-1249)

https://bugzilla.redhat.com/show_bug.cgi?id=2176140 (CVE-2023-1252)

So the Neowin headline is a bit misleading because Red Hat disclosed these issues publicly (presumably after consultation with the reporters) even before the 90-day timer expired in the Google bug tracker.

(Disclaimer: I work for Red Hat, but are not involved in security anymore.)

Additionally these are moderate CVEs and Red Hat, per their lifecycle page, patches them at their will.

During the Full Support Phase, Red Hat defined Critical and Important Security errata advisories (RHSAs) and Urgent and Selected (at Red Hat discretion) High Priority Bug Fix errata advisories (RHBAs) may be released as they become available. Other errata advisories may be delivered as appropriate.

Reading the article, they seem to be confusing CentOS and Stream... or intentionally trying to confuse the two.

Doesn't surprise me really. RHEL has priority for CVE's, always had, there is no reason for them to "test" CVE patches in CentOS Stream.

So this really speaks volumes about their commitment to CentOS.

RHEL doesn't have these patches either. Because they're local exploits and not all that important in the grand scheme - they're everywhere and you have to already be pwned for these to benefit an attacker.

Typically fixes like these would just be bundled together and wait for a regular release rather than be pushed out immediately.

I think Red Hat only receives priority for embargoed CVEs. Other CVEs are done first on CentOS Stream.

Rocky and Alma are vulnerable to these CVEs too, as is RHEL. As a matter of fact, CentOS Stream 9 already has the fix to one of the CVEs. The rebuilds have to wait for it to show up in RHEL before they can rebuild it.

What I don't see mentioned is that IBM would not have to back port anything from the 5.15 tree if they just used an LTS kernel in the first place. But instead they like to differentiate(?) their product by using non-LTS kernels and handling all the back porting themselves, rather than collaborating with distros like Debian and Ubuntu which use the LTS kernels.

I've wondered about this too. Like, what's the advantage of it?

It looks like they've gone back and forth on this. RHEL 7 and 6 used LTS kernels. Before that, RHEL 5 did its own thing instead of using the LTS. Of course, RHEL also supports these kernels for much longer than the original maintainers. RHEL 6 (released in 2010) still has another year of Extended Life-cycle Support into 2024, even though end-of-life for the 2.6.32 LTS kernel was in 2016.

Looking at everything now, I see that Ubuntu 18.04 and 14.04 also chose non-LTS kernels. Interesting.

Reasoning I've heard for this is since they backport complete subsystems and features from mainline kernel over the years, it turns into a frankenstein and it wouldn't resemble the origin version. So it doesn't matter if they'd used LTS version or a non-LTS one.

Ubuntu LTS doesn't always use an LTS kernel. 22.04.02 is in 5.19 I believe.

Red Hat doesn’t fix moderate CVEs immediately. Per there RHEL Lifecycle web page:

“During the Full Support Phase, Red Hat defined Critical and Important Security errata advisories (RHSAs) and Urgent and Selected (at Red Hat discretion) High Priority Bug Fix errata advisories (RHBAs) may be released as they become available. Other errata advisories may be delivered as appropriate.”

Seeing how all of these CVEs are considered Moderate they fall under the “delivered as appropriate”.

As usual people over react and try to make companies look like they don’t care about security, back ports, etc when all Red Hat is doing is following its own policy.

Absolutely agree, but one small point of clarification. One of the referenced CVEs (CVE-2023-1249) is rated low. The other two (CVE-2023-0590 and CVE-2023-1252) are rated moderate.

As of today Google still has not fixed the vulnerability on Pixel 6 a device that Google promised it would provide security fixes for 5 years disclosed by the Project Zero which is a Google suborg.

It has fixed the issue. I got the March update with the fix a few days ago.

> It has fixed the issue. I got the March update with the fix a few days ago.

The last Android fix for Android 13 for Pixel 6 is March 5, 2023. That's before the fix.

It was patched before the public disclosure, go read the security bulletin [1]. CVE-2023-24033, CVE-2023-26496, CVE-2023-26497, and CVE-2023-26498 are the big modem vulnerabilities.

[1] https://source.android.com/docs/security/bulletin/pixel/2023...

No, you're wrong: the OTA marked as 5th March security update has the fixes.

It's trivially easy to look up the list of fixes for Pixels, c'mon.

Maybe this will get Fuchsia going.

This is such a poorly written article.

There is an enormous difference between a vulnerability in CentOS Stream, and RHEL. The article title is about CentOS, but the body says RHEL is affected too.

Exactly, I followed the CVE link through, and all affected RHEL8/9.

Soo…… Red Hat isn’t patching their live, currently supported distributions? That seems bigger news than their approach to CentOS, which they are clearly trying to kill.

CVE-2023-0590 is already patched in C9, meaning the fix will likely will be shipped in RHEL 9.2. The other CVEs will probably get fixed as well (in CentOS first!), but when the RHEL maintainers are ready to, not based on some arbitrary deadline from a third party. Lower severity CVE fixes are routinely delayed until future minor versions, so this is nothing new, just an example of a security researcher not understanding how RHEL works.

You're obviously a Red Hat employee. It might be worth disclosing it.

I don't quite agree though. Sure, RH can decide when to patch, but a researcher isn't wrong just because they say "Hey, RHEL isn't patched".

Should RHEL be patching sooner? Maybe. Though I get patches can have unintended consequences. However, I like the idea of a third part scrutinizing this stuff. Otherwise companies will do the wrong thing and claim their security posture is perfect.

LTS is a dangerous concept, really, especially in the way Debian or Red Hat do it, that is, by freezing versions at a completely random point and then attempt to backport fixes.

This approach is outright dangerous, because in my experience a distro maintainer is often not as accustomed with the codebase as an upstream maintainer. I understand the need to keep versions stable, but I would rather use a real long term support version from an upstream maintainer than random patches from Red Hat.

That's why in my experience ironically rolling releases tend to be less finicky to use than long term support distros.

Is CentOS still alive? I thought it had been succeeded by Rocky Linux and Alma Linux.

CentOS 7 is technically still around, and Stream lives on as RHEL's beta channel.

Personally at my work I've been trying to navigate migrating our CentOS 7 production servers to EL8 (Looking at Rocky Linux) and it's been a bloody nightmare thanks to RedHat prematurely killing CentOS8. It seems like there isn't any kind of tools available on the internet to migrate off CentOS to another RHEL clone, so we may end up having to rebuild of our affected production servers before doomsday in 2024. Rocky has a bash script, but it only supports systems running EL8 or higher.

With RHEL, it has always been recommended to migrate your entire system to a new install of EL. They are rather notorious, historically anyways, for not providing their customers with an easy upgrade path.

This is one of the reasons we migrated away. It's pretty easy to go from 20.04 --> 22.04 without having to reinstall your OS.

I have taken long lived machines from 18.04 --> 20.04 --> 22.04. The idea that EL charges money with no in-place upgrade path is madness.

I basically learned to decouple the services from the OS it's running on (back before containers and all were a thing) so I focused on making it easy to spin up a new machine and restore a service. Over time, this made OS upgrades immaterial and made it easy to spin up an additional instance.

This led to containerization and virtual machines being a good fit for my habits.

Yup. I was working on a migration to CentOS ~8.2 before we got rugpulled. Took the opportunity to move to Ubuntu/Debian. If we have to do all this work, might as well change distro's.

AlmaLinux has ELevate to help you go from 7 to 8, and 8 to 9 coming soon.

https://almalinux.org/elevate/

This guy also helped find spectre/meltdown. I hope he writes a book or something to teach everyone his secret sauce

More details here:

https://packetstormsecurity.com/files/171409/GS2023032117320...

Local attack vector = nothing burger

Mainly because there are endless other local attack vectors.

Did anyone make the jump from 8 to Stream? How has your experience been? What's different?

Converted my homelab media server from CentOS 8 to Stream 8 early last year, haven't noticed an appreciable difference in stability or use-case. Stream 8 is EOL next year though, so I'm considering options of what to move it to.

At work an intentional decision was made to avoid migrating any prod/dev systems (previously on CentOS 7/8) to CentOS Stream.

Reply on news.ycombinator.com