Hacker News Reader

Comment by cthalupa | Hacker News Reader

>"The majority of the world runs Debian or they run their own kernel," Kroah-Hartman said. "Debian was not allowed to be part of the disclosure, so the majority of the world was caught with their pants down, and that's not good."

Is there any actual statistics to back this up? I feel like RHEL and to a lesser extent CentOS have a stranglehold on the big enterprise-y environments, and I see Ubuntu basically everywhere else, and Canonical does their own kernels.

Edit: To be clear, I am aware Ubunbtu is a Debian derivative, but since we're talking specifically about who was informed for kernel level mitigations, and Canonical does their own kernels, it seems weird to talk about how Debian wasn't informed and thus people were affected, when Ubuntu being updated wasn't reliant on Debian being updated.

View on HN

> The Linux distribution can be positively identified in around 30% of cases, and of these 1.39 million Linux computers, just over half are running Ubuntu Linux, nearly a quarter are running CentOS, and around fifth are running Debian Linux.

https://news.netcraft.com/archives/2017/09/11/september-2017...

There are a massive number of systems running Linux that are not publicly facing (e.g. compute/service nodes in supercomputers, systems for providing internal infrastructure for R&D, manufacturing, etc).

Such systems also most likely are not vulnerable to Meltdown/Spectre.

Why would you begin to think that at all?

Because they're offline doesn't mean that they're safe, just safe from more than an online system.

Those systems might be immune to attacks that networked machines are vulnerable to, but often networked attacks are used in order to leverage another exploit, such as Meltdown/Spectre.

A vector other than the network could be exploited to leverage an attack that utilizes Meltdown/Spectre.

Most the biggest software vulnerability catastrophies actually involve just such systems. "Secure internal network" is a 90s mirage. See eg https://www.wired.com/story/notpetya-cyberattack-ukraine-rus...

Meltdown & co. only become relevant once you can execute on a CPU. If an adversary can run software on your industrial robot, you are already compromised.

Think browsers and virtualization, and the various VPN's that invariably connect "internal networks" to the outside world via computers that straddle, or alternate between, other networks.

I think, what he meant by saying, that "the majority of the world runs Debian or they run their own kernel", – is that almost everyone else, who is not running a product by Red Hat, SUSE, or Canonical, is running a system based on a Debian kernel. And, therefore, Debian should have been informed.

That's very likely. I only posted that Netcraft survey because it was the only statistical data I know of :)

Even that interpretation contains little useful information. It's like some hobby-kit auto engine maker saying "The majority of drivers use our engine kits or they buy a brand name car." Sure, that might be true, but it doesn't say anything about how often that case actually is. In the example I put forth, it's obviously minuscule, and Debian is likely much more common that that, but how much?

It feels like someone at Debian is trying to avoid realy numbers or relative comparisons to make their case sound more important, but all it results in is statements like this where you look at it and go "WTF are they saying, because what it kind of sounds like seem highly unlikely..."

> It feels like someone at Debian is trying to avoid real numbers

As far as I'm aware of, Debian doesn't have any real numbers, because it's not an enterprise. However, GNU/Linux Distributions Timeline[1] gives a pretty good idea, how widespread Debian is in comparison to other Linux distributions.

[1] https://upload.wikimedia.org/wikipedia/commons/1/1b/Linux_Di...

If that actually denoted how common a distribution was in any way, we'd see a lot more Slackware in the world. Some distributions are much easier to fork than others, but a hundred forks of thousand users each doesn't compare well to four forks of a million users each, and the number of users is itself a nebulous concept (do we count installs, or people served by that install?).

I understand it's hard to count, but if they're going to make a case that your group deserved to be on the short list of groups that needed to know by calling on the prestige of their position, something a bit more concrete than some extremely vague statement like this.

Debian is very important and has a lot of prestige in Linux, due to it's principled stance on software and it's position as a base for other popular distros. Personally I think it's just not a type of importance and prestige that leads it to be as important to notify about these issues as the large enterprise distros (and their non-enterprise offerings). If you want to cover the most systems with the least amount of people involved, you hit the list Intel hit, and at some point you need to make a call on when to stop including people and groups. In the case of Debian, the type of contributors it attracts might actually work against it, as NDAs and threats of lawsuits work a lot better against people who are working under their capacity as a representative from their Company (even if as a kernel developer), and would face major consequences for breaking the embargo. Including random developers from across the world working in their spare time is much more dangerous, as there is no existing relationship with a company to fall back on to vet the individuals, and the negative consequences for breaking the embargo for those individuals might be quite small, depending on circumstance. I imagine it's quite a bit more work for Intel's legal department (possibly in background checks they don't even want to think about having to figure out) to sign off on what might be the suggested people to include from Debian.

> If that actually denoted how common a distribution was in any way, we'd see a lot more Slackware in the world.

According to the graph, Slackware is a way less widespread than Red Hat, and Red Hat is a way less widespread than Debian. It seems to be a correct representation of reality, given the popularity of the .deb package format, and the size of apt package repositories, in comparison to .rpm and yum, respectively.

I agree, that Intel might have chosen not to inform Debian because of its non-enterprise nature, but it doesn't change the fact, that Debian might be the second most popular Linux server distribution in the world[1], with the market share 20x of that of SUSE, which was informed about the bug by Intel ahead of time.

[1] https://w3techs.com/technologies/details/os-linux/all/all

> According to the graph, Slackware is a way less widespread than Red Hat, and Red Hat is a way less widespread than Debian. It seems to be a correct representation of reality, given the popularity of the .deb package format, and the size of apt package repositories, in comparison to .rpm and yum, respectively.

As I noted before, this graph is just showing a family tree of distributions, not use. If I made a hundred different different distributions they would show as a hundred different branches here. That wouldn't mean anyone actually ran or runs them.

"Widespread" only really makes sense with respect to install or end users, neither of which are represented here at all.

> Debian might be the second most popular Linux server distribution in the world

That's not what your link says. It says it's the second most common distribution for powering websites (so it doesn't count dedicated mail servers, cache servers, database servers, file servers, etc). It also says that "Unix" is almost twice as popular as Linux under the same methodology. That likely means BSD variants, but whatever methodology is showing that ~70% of all websites are running on Unix seems somewhat questionable to me.

Are you claiming that there is not enough evidence of Debian being a major Linux distribution, whose developers should be informed of major security bugs before they become public?

> It says it's the second most common distribution for powering websites

There is no other way to measure and compare the popularity of open-source operating systems. Any search for "operating systems market share" shows Debian only seriously behind Ubuntu, and close to CentOS[1][2].

Do you also question the popularity of Ubuntu and CentOS?

[1] https://www.datanyze.com/market-share/operating-systems/debi...

[2] https://www.similartech.com/categories/operating-system

> Are you claiming that there is not enough evidence of Debian being a major Linux distribution, whose developers should be informed of major security bugs before they become public?

I'm claiming that Debian's wording was somewhat weird, and that they aren't necessarily as large as it makes them sound.

Additionally I'l claiming that the nature of Debian itself may make it harder to get them vetted for access (I suspect the kernel developers that got access are employed to work on the kernel by large companies, a fairly common scenario for Google and Redhat for example, which is another point for including them).

Also, I'm stating that public numbers got operating system by website is a very poor metric, which I'll cover below.

> There is no other way to measure and compare the popularity of open-source operating systems. Any search for "operating systems market share" shows Debian only seriously behind Ubuntu, and close to CentOS[1][2].

Just because only some numbers are available does not mean they are worth using. For example, in the SimilarTech link you provided[1], you might notice that the operating systems usage percentages shown for the top 1k sites add up to less than 5%. For the "entire internet" statistics it's just over 1.5%. I interpret this as the vast majority of sites having no discernible data as to what operating system they run (for their testing criteria, at least). I suspect it's really a measure of servers started that have the default HTTP server running with a landing page saying "Thanks for installing CentOS" or has some poorly configured protocol with an OS identifier in it (which has been deemed poor security for quite a while, which might explain ancient Unix systems having a large showing...).

For the Datanyze stats[2], the total websites shown add up to just over 2.1 million. What we don't know, since I can't find any info on their methodology, is whether they suffer from the same problem I suspect SimilarTech does, in that they are reporting vastly biased data because they can only determine the OS of a small subset of the total sample.

Additionally, this doesn't count desktop installs. Google alone likely has over 20k developers[3], and the default desktop for them is Goobuntu (a Google variant of Ubuntu). While I expect Windows and OS X to have a large share of that market overall, the relative number of desktops compared to servers means that it may affect the numbers greatly.

There is actually a way to semi-accurately measure operating system usage, at least for operating systems that use default update mirror-lists and auto-update (most linux distros, to my knowledge) randomly (or using some specific leveling algorithm) spread requests along the mirrors, if you could get stats on actual downloads of a core package that had a security update within the last 6 months to a year (preferably daily or months stats) for a few mirrors, you could statistically derive a lot of information are active installs. You'll miss people that auto-install with special configurations that point to internal mirrors (common in some business settings), but you might get within 10% of a real number for a distribution.

> Do you also question the popularity of Ubuntu and CentOS?

I question any number presented by these sites. I have plenty of anecdotal evidence to suspect they are popular. If I did not have even that anecdotal evidence, I wouldn't assume to know much from the data we have seen so far.

1: https://www.similartech.com/categories/operating-system

2: https://www.datanyze.com/market-share/operating-systems

3: https://www.quora.com/How-many-software-engineers-does-Googl...

> I'm claiming that Debian's wording was somewhat weird, and that they aren't necessarily as large as it makes them sound.

In fact, it was not Debian's wording, but Greg Kroah-Hartman's, who is currently responsible for the Linux kernel -stable branch, and who has previously worked for SUSE, where he initiated the development of openSUSE Tumbleweed[1]. So, he is in no way associated with Debian, and is one the most important employees of the Linux Foundation.

> Additionally, this doesn't count desktop installs. Google alone likely has over 20k developers[3], and the default desktop for them is Goobuntu (a Google variant of Ubuntu).

Interestingly, Google ditched Ubuntu for Debian in January 2018[2], which means, that the entire Google internal engineering environment is now based on Debian.

[1] https://en.wikipedia.org/wiki/Greg_Kroah-Hartman

[2] https://www.theinquirer.net/inquirer/news/3024623/google-dit...

> In fact, it was not Debian's wording, but Greg Kroah-Hartman's

Ah, thanks for the correction. I was under the impression it was Debian speaking. I still think it's an odd way to describe the group in question (those running vanilla kernels), but there's less negative connotation since the obvious benefactor of that wording wasn't responsible for the statement.

That said, it's clear that some of my prior statements were erroneous.

> Interestingly, Google ditched Ubuntu for Debian in January 2018[2], which means, that the entire Google internal engineering environment is now based on Debian.

Interesting. I wonder if they did an easy-upgrade for the Goobuntu installs, or if it's just for new installs. To my knowledge, Google has a recommended Linux distro, and developers can choose it or choose another supported distro if they have reason (i.e. There are devs running windows and OS X). In that type of situation, I would think some portion of developers might have switched quickly, others as they had problems and needed a reinstall, and finally new hires. There's probably a good mix of the two distros, which will shift towards the new one consistently over time. It will undoubtedly increase install numbers though, as it's not a small amount of people.

I still wish we had some better numbers on this. :/

The Debian popularity-contest package can give you an approximate lower bound, though since it's purely opt-in it's very much an underestimate.

Honestly I didn't even know suse still existed, Linux distros are dominated by Debian/Ubuntu and redhat/centos

The majority of Linux installations is probably Android smartphones, and well Google was informed alright, but still 95% of those phones were caught with their pants down because well their pants had fallen down a long long time ago given they last received security updates years ago.

How many android installations use Intel processors?

Edit: never mind, Android was apparently indeed effected.

https://www.androidcentral.com/meltdown-spectre

Spectre affected virtually every out-of-order execution CPU in existence; Meltdown was much more limited in scope (and very few non-Intel CPUs were affected, but some were).

Spectre and Meltdown are a problem for Android but it's a smaller problem than on PCs. The vast majority of Android phones use processors that are in-order, and lack the out-of-order execution necessary to exploit.

High-end Android phones with out-of-order processors (the minority) are only vulnerable to Spectre, but it's a little complicated [1]. Then there is only one ARM core (Cortex A75) that is vulnerable to Meltdown, but that was released after the vulnerability was published, so software could ship with mitigations in place.

[1]: https://github.com/lgeek/spec_poc_arm

Some of Apple's A-series processors were affected by Meltdown too, AFAIK, though obviously that doesn't affect Android. (But is another ARM core vulnerable to it.)

I can speak to the companies I've worked for: US companies (and, this includes the UK because apparently we love to be like america) use CentOS predominantly, but mainland europe (Sweden, Germany, Finland) seem to prefer Debian.

I can say that I've personally administered roughly a 1:1 ratio of CentOS:Debian despite coming from a country that's servers tend to be CentOS.

Of course, this is anecdotal, but don't undersell debian.

Is SuSE still popular in Europe? (Apologies if I got the capitalization wrong)

Besides SAP deployments I guess not much.

Here in Germany it is mostly Red-Hat/CentOS or Ubuntu, on the projects I have been involved.

So anecdote data.

Since 2008 working for US/UK companies I have only seen RHEL and CentOS (prior to that, mostly SunOS/Solaris). I prefer Debian personally but for the last few years I have used CentOS for my own projects too - too much effort to maintain two parallel sets of skills now I am not really into OS level stuff anymore.

https://youtu.be/lQZzm9z8g_U?t=1565 seems to be the same talk delivered at another instance of this conference. He does claim one statistic that an unnamed "top 3" cloud provider told him that fewer than 10% of their customers install company-based kernels and over 90% install community-based kernels like debian or kernel.org code.

He is not limiting "the majority" to just Debian, though. He is comparing installs of distributions backed by companies (like Red Hat+Canonical+SuSE) as "the minority" vs. every possible non-corporate-backed distribution or source.

By "majority" he may have meant majority of Linux distributions, not majority of Linux users. There are a ton of distributions derived from Debian: https://wiki.debian.org/Derivatives/Census . All of them would have to update their distros to respond to Meltdown.

Perhaps Greg-KH was implicitly including Ubuntu under Debian?

Yea I don't use Linux too often but how different is an Ubuntu kernel compared to a Debian kernel? I know Ubuntu is downstream from Debian or at least they were. So how much extra work/code does Canonical add to its kernels.

Canonical had Spectre and Meltdown mitigations out for their kernels before Debian did, and he also specifically mentions Canonical as getting alerted, so apparently enough.

I think there should be a distinction between Canonical and debian, but Ubuntu is certainly based on Debian, and the latter should be used when talking big scale like he did. Many cloud services and providers run on Ubuntu, and as long as Ubuntu uses apt and .deb, it's Debian (along with Linux Mint, elementaryOS, etc)

> and as long as Ubuntu uses apt and .deb, it's Debian

No, there is more to a distro than them sharing a common package manager/format. The kernel, which is heavily patched by canonical, differs from what Debian ships. Canonical also has a bad habit of carrying a lot of non-upstream patches for other core system components. Ubuntu may have been derived from Debian at one point, but they are fairly different at this point. So much so that you cannot take a system running Debian and 'upgrade' (or downgrade, depending on your point of view) it to one running Ubuntu.

Actually, you can smash Debian & ubuntu together, you just get a terrible monster called frankendebian. As a youngster I tried this, it didn't pan out.

A distro, for the most part is 1) package manager + repos, and 2) system organization (where things get installed, where configs are, how services are managed etc). In that respect, Ubuntu is still Debian - most packages are vanilla Debian packages, and they both use systemd.

> So much so that you cannot take a system running Debian and 'upgrade' (or downgrade, depending on your point of view) it to one running Ubuntu.

That doesn't say much, as sometimes ubuntu and debian upgrades are far from smooth.

As a long Debian & Ubuntu user, I believe you're exagerating the work invested by Ubuntu to custonize Debian. Betond the default desktop environment and attitude regarding proprietary drivers, the work done by Ubuntu is rather negligible.

Canonical was alerted, but they still had a delay of several weeks before they had an update.

That's not true.

- https://www.debian.org/security/2018/dsa-4078 (January 4th) - https://usn.ubuntu.com/3522-1/ (January 9th)

Canonical got caught in the shortened embargo and didn't modify their schedule while everyone else did.

As mentioned in the article, Canonical were brought in much earlier by Intel. They appear to have been among the "chosen few": RedHat, SuSE and Canonical.

I'm sure he cares a lot about Debian and a lot of my early learning was on Debian derivatives but I feel that the production server side world has largely standardised on RedHat. Non RedHat/CentOS, non-Ubuntu debian distros are probably a distant fifth in terms of business critical production server so no wonder they were not included in the disclosure early on...

Does CentOS get immediate updates like this? My overwhelming impression of that platform is that everything is years out of date... of course maybe that's just the fault of the organizations I've encountered that have used CentOS.

Yes it does. Both RHEL and CentOS have a kernel that is "officially" 3.10 but feature-wise is probably something like 4.10-ish for RHEL 7.5 (which was released last April).

RHEL and CentOS 7 had updates for all of Spectre, Meltdown and Foreshadow on the day of unembargo-ing.

I'm pretty sure it's being kept fairly up to date since Redhat acquired it. Bear in mind that it's a RHEL clone so it's usually about as out of date as RHEL is - which is what we generally want in the infrastructure/telco/banking space because stability is preferred over shiny new features.

CentOS gets patches as soon as Red Hat publishes the source RPMs upstream or pushes changes to git.centos.org.

What can be confusing about CentOS/RHEL is that many shops using them are on a release that came out years ago. But that's an issue because Red Hat supports 10 year lifecycles for their releases, and will go on supporting them indefinitely if you pay them enough. All the software bundled gets feature and security updates through it's lifecycle, just without changing any interfaces, ABIs, or version numbers. This is great for shops that aren't looking to operate on the bleeding edge and have expensive COTS or bespoke software stacks sitting on top.

Of course, you still have to patch, and organizations can't be forced to do that. Hopefully what you were encountering was just the former and not the latter at those organizations.

What you're likely seeing, and what the siblinb comments are referring to with the "support" you get for years, is that RedHat (and CentOS by extension) do "back-patching", which is to either take the existing patch diff and alter it to work with a slightly older version of the software, apply it and shit that as an update (with the same software version number but increased release number), or to create a wholly new patch to fix it if required and do the same.

This allows you to have a stable "base" with unchanging software requirements, configuration and features to work on for extended periods while also staying very secure, which is why it's favored by enterprises. You can find more information on this on their Life Cycle documentation.[1]

This behavior actually extends to the kernel itself, which is likely why the Debian developer is quotes talking about "the majority of the world runs Debian or they run their own kernel", as Redhat, Fedora, Ubuntu, etc run custom kernels that they've patched. You can see this in different bugfixes for a package, and how they have the same version number for the software, but different release numbers, which come after the version in the filename.[2][3]

There are some more nuances to this model, where you can actually get newer software versions at select points, or where they will actually backport new features and not just bugfixes, but those are somewhat separate, and happen much less frequently (point releases in RHEL terminology).

1: https://access.redhat.com/support/policy/updates/errata/

2: https://access.redhat.com/errata/RHBA-2017:2576

3: https://access.redhat.com/errata/RHBA-2017:2926

I run centos in production. That's because of the following:

     1. SELinux 
     2. OpenSCAP
     3. 99% fit with RedHat

None of the other Linux variants have those tools, or the time put in for compliance goodies. My time == $$$

Why would an off-the-record statement, which is what he said this was, need to be 100% without exaggeration or hyperbole?

The article doesn't mention it was off the record, and I wasn't there, so how would I know it's off the record?

But it seems likely that this isn't just exaggerated or hyperbolic, but just not close to accurate, in which case even if it's off the record it's silly.

Ubuntu is a Debian derivative[1], I imagine that is what he means:

[1] https://www.debian.org/derivatives/

Yes, but the problem is we're talking specifically about kernel level mitigations, and Canonical rolls their own kernels. They do NOT use debian kernels.

Please re-read the context. I was simply explaining to the OP why this individual may have said this.

That doesn't make sense, though. Debian and Ubuntu each roll their own kernels. Canonical was one of the chosen few that had advance warning from Intel. Debian did not, so it wouldn't make sense for Greg K-H to be talking about Ubuntu at all when he referenced Debian.

Translation: s/The majority of the world runs/My fellow Debian kernel developers run/

Desktop and personal PC has Ubuntu dominant, but servers are 70% CentOS. Nobody trusts Debian's security record and gcc/kernel abilities.

He is right with the affected number in this case because this time bugs are affected by simple webpage drive-by Javascript attacks, like a common virus. So it affects Debian and Android more than servers and cloud services. And those are the ones which are not regularly updated so the security impact is bigger than on a server.

Reply on news.ycombinator.com