Linux-Kernel: Printk() Indexing | Hacker News Reader

Linux-Kernel: Printk() Indexing

lwn.net

80 pointssigna1115 comments

I mean this semi-seriously but doesn't the hypothetical example "CPU on fire" vs "CPU in flames" violate We Do Not Break User Space? As someone who got a useful (IMHO) one-line patch with less change dinged for WDNBUS, I would say yes.

According to the article "those messages have never really been considered to be a part of the kernel's ABI"

Right, but the We Do Not Break User Space rule has often been explained (somewhat emphatically on occasion) as depending on whether anyone's code actually gets broken, not on whether the thing they were relying on was intended to be, or advertised as, a stable interface.

I think WDNBUS is only respected for the syscall api.

What about /proc ?

/proc has surely killed some user apps. If you were expected to see a fixed number of lines, a fixed number of columns or the fields to mean the same thing, you're in for suffering. At least I know of meminfo changes, stuff in /proc/net...

But to be honest, so have ifconfig, hostname, df...

At least in procfs they add things (columns, lines) so apart from meminfo I don't know of changing the meaning of fields.

Here's a relatively mild rant where the breaking change in question was in /proc/cpuinfo:

«Really. Shut up.

The whole "no regressions" thing is very much about the fact that we don't waste users time.»

https://linux.kernel.narkive.com/7hMPh8zX/patch-revert-9fc21...

Of course in an ideal world the extent of the intended promise would be clearly written down in some obvious place. But we don't live in an ideal world.

I found this statement from Linus in that thread;

"It's not that the kernel ABI (with all the system calls, all the /proc files, all the ioctl's, etc) is set in stone and "sacred". Absolutely anything can be changed, wildly.

But if it turns out that applications (or hardware) that people use end up breaking noticeably, then that is a regression."

Linus has been pretty strict in not unnecessarily breaking userspace even if something is not strictly ‘part of the ABI’. It’s a judgement call though.

>The 2011 Kernel Summit featured a combative session around a proposal to add a 128-bit binary tag to every kernel message. That idea, like all others before and after, failed to get anywhere near the mainline. The kernel community has, quite simply, never felt the need to impose any structure on the logging information it puts out via printk(); the additional effort that would be required does not seem to be worth the cost.

It seems like that was the more elegant solution vs enabling debugfs, running through the list of error messages to ensure nothing changed. How many distros even ship with debugfs enabled by default?

Edit: formatting

I think the problem then becomes what message gets what tag and you have to bikeshed over if smaller or larger numbers mean bad/urgent messages. It's like the same problem of port allocation that IANA was created to solve. Do we really expect every kernel module to adhere to e.g. registering a unique tag with some authority for their messages? What if two separate modules print the same tag? And so and so on, and all this time of arguing over what is the "Linuxy" way to match numbers with format strings would be better used reviewing commits that could actually improve the kernel.

I think the better morale of the story is to not rely on third party log output to troubleshoot problems. If you want to be notified when the CPU catches fire then you should have a daemon watching CPU temp waiting to trigger the event, not relying on some black box to log the correct line at the right time and some other black box to match the line against some ancient regex that nobody understands anymore. At the very least your temperature monitoring daemon should `printk` a message you decided upon instead of someone else.

> How many distros even ship with debugfs enabled by default?

Almost all of them, try `mount | grep ^debugs` or at least `/proc/filesystems |grep debugfs`. It's root-only though.

If anyone wants to steal a done-tested-and-working userspace variant (for C, and with limitations C++) of this:

https://github.com/FRRouting/frr/pull/6766

It's pretty much the same thing, all occurences of "zlog_debug" and the like create an entry in a separate ELF section. The section can be accessed both at runtime by the program itself as well as offline by reading the appropriate ELF headers (there's an extraction tool in the codebase that does that, it even works cross-OS cross-CPU cross-endian).

disclaimer: it's my code ;)

disclaimer[2]: this will not work without major overhaul on OSX or Windows. It relies on the binary format being ELF.

disclaimer[3]: the PR linked above is only the "core" of it, you may have to fish out other pieces elsewhere.

The article mentions debugfs which I've never heard about and is pretty cool: https://www.kernel.org/doc/html/latest/filesystems/debugfs.h...

Debugfs is just a dumping ground for various debug functions. For instance ftrace subsystem is there. Of course, it's 700 root by default, because you can find a lot there.

Reply on news.ycombinator.com