Warning: Debian stable kernel upgrade breaks most ARM SBC | Hacker News Reader

Warning: Debian stable kernel upgrade breaks most ARM SBC

bugs.debian.org

132 pointsgeppetto37 comments

Oops: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=922478#72

The problem patch is in http://deb.debian.org/debian/pool/main/l/linux/linux_4.9.144... (I think)

"Commit 901e325f772f "ARM: bugs: add support for per-processor bug checking" added a new member to struct processor. This structure is always instantiated in built-in code and the new member is only used in built-in code. Therefore we can safely add the new member at the end instead.

Move it to the end and hide it from genksyms. Also hide it when building modules, to make sure they really don't use it."

This should be made "sticky", if there was this feature @HN. Strange that ci.debian.net didn't catch this.

What % of HN users use an ARM SBC with Debian?

I'm sure it's small, but at least a few of us are here. Also worth noting is that (most) Raspberry Pi (an ARM SBC) users are indirectly Debian users too. (and I'm a Pi user as well)

My BeagleBone Black runs a Debian-based distro (Angstrom), and there are spins based on Debian proper, Ubuntu, etc. So it’s definitely good to know.

The BBB doesn't use an upstream kernel though as far as I can tell.

Ångström appears to be based on OpenEmbedded rather than Debian:

https://en.wikipedia.org/wiki/%C3%85ngstr%C3%B6m_distributio...

There's just about zero chance that this bug would make it into raspbian stable, though, since they are always at least several months behind debian-stable on AMD64.

And you'd really hope they'd actually test it on real hardware. Although I don't know if they have anything automated, that'd be interesting to know.

Not Debian directly but one of their derivative: Armbian, they have great support for Allwinner SoC and it's easy to build your own image.

We do at https://fosdem.org on our custom made video gear. That's 56 boxes in a crucial spot in our operation. We went out of our way to make sure we had plain vanilla mainline Debian stable packaged linux running on the Allwinner A20 based boards that power them, instead of some binary arm kernel with all kinds of issues.

Was your fleet impacted by this?

Edit: oh, FOSDEM was 2 weeks ago, not in the near future, so impact is less

OffTopic: At FOSDEM, I was quite curious about those media boxes. Is there any docs to read about them?

Here's some info at least:

* a rough diagram: https://github.com/FOSDEM/video/tree/master/hardware

* the software running on them: https://github.com/FOSDEM/infrastructure/tree/master/ansible...

Your message also served as a reminder that I should really publish the laser cut box cutting and assembly info. I'll get to that eventually.

Off-topic, but Fosdem is great! Started going about 6ish years ago and always enjoy it!

Thanks for the hard work and for providing such a great event for free. And for the cool t-shirts :D

Thank you for the kind words. Much appreciated!

I have a Rock64 NAS box running an RK3328 chip.

I do, at least for some personal projects. They're more than capable enough for many tasks, and cheap enough to be willing to just grab another if you have a quick task.

I don't run Debian, but I have multiple ARM SBCs that I work with as well.

I have 10 Raspberry Pi's with Debian Stretch and Jessie - the Raspbian variant. I updated a "sacrificial" Pi and rebooted without error.. this time!

  Package: linux-libc-dev
  Version: 4.9.82-1+deb9u3+rpi1
  Priority: optional
  Section: devel
  Source: linux-4.9
  Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org>
  Installed-Size: 4,466 kB
  Provides: linux-kernel-headers
  Homepage: https://www.kernel.org/
  Download-Size: 1,300 kB
  APT-Manual-Installed: no
  APT-Sources: http://raspbian.raspberrypi.org/raspbian stretch/main
  armhf Packages Description: Linux support headers for userspace
  development This package provides userspaces headers from the Linux
  kernel.  These headers are used by the installed headers for GNU libc
  and other system libraries.

edit: formatting

As others have mentioned, Raspbian kernel updates are delayed behind the Debian ones. The linked bug report states it was introduced between 4.9.135-1 and 4.9.144-1 (of linux-image-4.9.0).

I do, but I'm OK with being a statistical fluke.

I imagine there could be a number of IOT, media or other devices unknowingly running Debian on ARM though; more than we realise.

I use 2 Olimex A20 Micro with Debian stable as personal servers (Web, Email, Backup). One of them had already installed the toxic update via unattended-upgrades. Fortunately I did not configure automatic reboot. So I had the chance to read about the issue here on Hacker News and reinstall the old version:

$ wget https://snapshot.debian.org/archive/debian/20181028T150508Z/...

$ dpkg -i linux-image-4.9.0-8-armmp-lpae_4.9.130-2_armhf.deb

and set the version on hold:

$ apt-mark hold linux-image-4.9.0-8-armmp-lpae

I'm glad I can. It's a known, trusted distribution I've been using since 2001 on my main PC. It runs my personal web server since 2013 on a Cuibeboard and it works well. I've got a stable system which I'm familiar with, plus automatic updates, no binary blobs, vanilla kernel and all.

I can't think of a better solution if the SBC is supported. What would you recommend as an alternative?

As far as I can tell, ci.debian.net doesn't support any of the ARM architectures and does not test suites other than testing and unstable:

https://ci.debian.net/data/

Also, I think something like LAVA is more suited to kernel/bootloader testing than debci:

https://wiki.debian.org/LAVA

Why do they call it "stable" then?

Because when it breaks, it is broken in a stable way.

This is actually 100% true. Stable is the culmination of the Debian Release Engineering / Testing Freeze process.

In Unstable, packages are promoted from Experimental as I understand it which is a comparatively easy threshold to get past. In Testing, packages are only promoted from Unstable if they 1) don't have any bugs open against them, and 2) there is not a freeze on. 50% of the time (1 year) there is a freeze on.

In Freeze, only bug fixes are promoted to Testing. New features have to take a number.

When the freeze is over, Testing is cut and becomes the new Stable. I might have described some part of this subtly wrong, but this is the Debian release management process in a nutshell. To read more: https://release.debian.org/

In short, "Stable" should be read like stable compound, not like we usually interpret it to mean "doesn't crash" in IT.

It's a side effect of not changing often, that it probably doesn't crash too much. The crashing bugs wouldn't have made it into testing, or through the testing/freeze process. Hopefully, at least.

I personally prefer to run Unstable on my developer machines, because it gets fixes more frequently:

> Security updates are made by the maintainer; they may not be effective on all architectures, and may be delayed. Packages uploaded may not meet release standards, but any breakage is expected to be fixed promptly. Updates are made by maintainers.

>It's a side effect of not changing often, that it probably doesn't crash too much.

I don't think that's true, it's likely that the actual causes of crashes are better known and documented, I don't think there is a particular way you can make a software release be less likely compared to other releases of the same software.

The "big or disruptive changes" have to go in sometime, and crash bugs are often documented before they are fixed...

> If you are planning big or disruptive change, check the timeline to see if it's still realistic to finish them before the transition freeze.

This might have been a better link to post. The freeze happens in stages "transition freeze / soft freeze / full freeze"

https://release.debian.org/buster/freeze_policy.html

Anyway I phrased that all wrong. I meant to say, that if it doesn't crash as much for you, you can expect that to stay the same throughout the stable release. The changes are universally where the new bugs are coming from. So it follows that you should expect less new bugs, because there are so few changes allowed in the stable distro.

Security fixes and Updates are delivered separately. I don't know this process inside and out but I've been exposed to it for 20 years, here are some more good current links:

https://wiki.debian.org/StableUpdates

https://wiki.debian.org/StableProposedUpdates

https://wiki.debian.org/DebianReleases/PointReleases

> Even stable is updated once in a while. Those updates are called "Point Releases". They usually incorporate the security fixes released until the time of the update and fixes for grave bugs in the current release.

Misleading label to say, e.g. "The horse is unstable." when intending to explain how the horse is outside the stable. (re: Stable == Compound in Debian-speak.)

Hmm, I meant stable compound in the sense of a noble gas, not highly volatile or reactive. Or say, lower energy states that are not as likely to emit a photon.

The horse is "unstable'd", maybe another way of saying it. Unstable is developed outside of the stable release engineering process. That does not mean it's without safeguards. The unstable releases tend to be very stable. There are extra safeguards, and time-based safety measures that protect users of the Stable distribution. They say that Unstable might take longer to get fixes, and while that's likely true in the case of critical fixes with special attention, it's almost never true in practice other than that. Unstable receives fixes much more quickly, in general.

There actually used to be a "debian-volatile" project, too, for things like virus definitions that should be used in a stable distribution, and also didn't make sense to govern through the stable release process, but it is defunct now.

> "Because when it breaks, it is broken in a stable way."

Hubris.

How can you declare something "stable" if you don't perpetually run tests to verify it is indeed "stable"?

And to state the obvious: certain bugs will lead to instability, non-deterministic, and undefined behavior - that's the very nature of things breaking down.

When Debian says "stable" they mean "we don't upgrade the software significantly". Ie, the system doesn't significantly change.

Of course that means it's about as stable as in not crashing as any other distro.

love this note on it: "Sure, maybe. I've suggested kernelci as a useful thing to help here, but we really need to be testing kernels complete with all the Debian patches to..."

so does that mean they test the kernel but not their own patches to it? seems silly :D

then it's not a debian stable kernel now is it

Reply on news.ycombinator.com