Hacker News Reader

I agree! For Windows, getting symbols was trivial: you just pointed your debugger to the publicly-available symbol server[1]. You can see this attitude continuing in [2]: a beginner is warned away from using a debugger to understand the kernel behavior.

[1] https://msdn.microsoft.com/en-us/library/windows/desktop/ee4... [2] https://lists.kernelnewbies.org/pipermail/kernelnewbies/2016...

Wow, that second example is very telling. For whatever it's worth, before we started work on DTrace (ca. 2001), we prioritized a project to add not just symbol information but also debugging information to production kernels. This project -- the Compact C Type Format (CTF)[1][2] -- has proved essential many times over in the years since (and is a major reason why we can consider kernel debugging a part of the core system functionality). It is clear that a project like CTF is unlikely to even be understood let alone prioritized by those displaying such a dismissive attitude towards debugging.

[1] http://illumos.org/man/4/ctf

[2] http://www.smnd.sk/lovasko/paper.pdf

Brian, I love your work but seriously. You've mispelled "read" as "understood" there and to write it off as a dismissive attitude toward debugging in toto is not something you should lower yourself to pretend to believe. Deep breath. DTrace is awesome, Oracle is not.

Yeah my kissing may not be up to scratch either... ;-)

The attitude towards debugging is dismissive -- and I'm not the only one to have drawn that conclusion. Indeed, it's the original author of this article who came across that and drew that inference -- one that I (obviously) share. To flip this around: do you think that the work outlined in the original article is work that should be expected of anyone wishing to debug the kernel?

I think there is room for reasonable people to disagree reasonably about most issues. Flip that around?

Linux is a long way from being the buggiest OS kernel I've ever used, how about you? Perhaps they've found and fixed some bugs? Perhaps they've prevented some from being written? Perhaps their approach is something that can be disagreed with, even strongly so, on the grounds of being less than optimal without suggesting that it has zero merit and by extension its proponents are somehow to be considered with derision? The inference that anyone hacking any OS kernel is too stupid to understand a differing idea is probably not necessary and unlikely to be justified in my humble opinion. You may of course reasonably disagree and maybe one of us did understand something the other did not on the point? Anyway this is now dull.

But don't be less opinionated, that would be the wrong response!

> Perhaps their approach is something that can be disagreed with, even strongly so, on the grounds of being less than optimal without suggesting that it has zero merit

If a yes or no attitude results in fewer bugfixes, all else equal, then that attitude does have exactly zero merit.

Errm, the email at https://lists.kernelnewbies.org/pipermail/kernelnewbies/2016... seems to be warning him against using an unusual set of custom compiler optimisation options that none of the kernel developers have ever tested with rather than against using a debugger full stop.

The beginner wants to disable optimizations, so he can better understand the kernel behavior by stepping through it in a debugger; he's told this isn't possible. Compare this with the availability of checked builds of Windows components, which support debugging by disabling optimizations: https://docs.microsoft.com/en-us/windows-hardware/drivers/de...

Checked builds are no longer distributed.

Source: https://social.msdn.microsoft.com/Forums/windowsdesktop/en-U...

I think checked builds did not disable optimizations, though it did include useful stuff.

That's a great Torvalds emails which I haven't seen before! Here's the "point" though:

> I happen to believe that not having a kernel debugger forces people to think about their problem on a different level than with a debugger. I think that without a debugger, you don't get into that mindset where you know how it behaves, and then you fix it from there. Without a debugger, you tend to think about problems another way. You want to understand things on a different _level_.

I think that's really pretty reasonable. Surely you've seen a programmer who ran into a bug and submitted a pull request saying "I fixed it" but it's clearly just a messy work-around which misunderstands the problem and leaves a bunch of analogous problem cases unfixed.

I prefer printf-and-inspection debugging myself, I only reach for gdb once a week or so. I'm pretty diligent and my results are quite solid. It's fine that we have different styles, and because of well-designed interfaces, like process boundaries, we can co-exist in peace.

Linus is really counter-cultural in today's hip software world, where everyone says everything should be easier and easier. We just can't ever make threads or async or crypto easy enough. Meanwhile the popular applications and frameworks in use seem to get bigger and slower and buggier. Correlation, not causation, but still an annoying trend.

> he has conflated single-step in situ debugging (which does in fact suffer from limited utility in the context of an OS kernel) with debugging writ large.

Where does he do this? The quote is only about single-step debugging.

> So as he rejected in situ debugging, he also implicitly rejected postmortem debugging and dynamic instrumentation

Do you have any evidence of this? It seems to me that dump traces and kernel probes have been an important part of Linux for many years.

Not to mention relocatable kernel and kexec crash dump analysis.

No, the quote is not only about single-step debugging; his next sentence is: "I do not think that extra visibility into the system is necessarily a good thing." And if that still leaves too much ambiguity for you as to his thinking, read the rest of the screed -- and then remember how we got here: because it took someone two painful days to configure something that is considered core functionality on many other systems.

It's a lot easier to configure using the modern API - the author is intentionally choosing an API which doesn't give much feedback....

That's true, but the author has solid reasons for doing so:

> I want my application to work on OS X and Linux, so I’m targeting PF_KEYv2 instead of OS-specific APIs.

If the reason he's using a horribly unfriendly and difficult to debug API is because it's cross-platform and the more developer-friendly replacement is Linux only, then the fact that the API is so hostile certainly isn't proof - as bcantrill was claiming - that Linux is somehow going out of its way to be hostile to developers compared to competing OSes.

PF_KEYv2's difficult interface definitely isn't a Linux issue. But PF_KEYv2's limited error reporting means that understanding what's going wrong requires kernel debugging. The difficulty of kernel debugging is a Linux issue. ("I do not condone single-stepping through code to find the bug.")

[1] https://news.ycombinator.com/item?id=15953644

This is Linux after all, why would you need postmortem debugging if the system never kills itself when it finds itself in an invalid state? :-)

For what it's worth, Microsoft has the same approach to debugging the kernel via WinDBG.

You can debug the kernel locally using windbg, it's pretty easy to freeze your system doing that though.

How do you suggest a kernel debugger should work?