Hacker News Reader

Comment by thegeomaster | Hacker News Reader

Maybe it's worth emphasizing that writing kernel code is a very responsible task, not to be taken lightly. Hackers coming from a userspace background tend to think it's just "userspace with a few quirks" which is grossly wrong: your code has to be completely free of bugs, as a dangling pointer, for instance, won't just send you a friendly SIGSEGV and terminate your ass, it will crash the whole system, leading to consequences for everyone who relies on its stability. Just imagine introducing a bug which crashes a Linux-based life support system. You could be responsible for endangering human lives. (I ageee that Linux isn't a sane choice here, but just for the sake of the argument.)

Also, standards to get your pull requests merged into mainline Linux are insanely high, but even given that, you can never rely solely on maintainers' code review to catch your mistakes. Take kernel hacking seriously.

View on HN

That's putting it a bit harsh. Yes, if you plan on writing kernel code for your fleet of production systems, or for submission to mainline, you need to be very careful.

That said, starting out, you work on a development system, you write a bunch of code, it'll have a lot of bugs, and you'll learn from them and from the process of fixing them.

Saying "You could be responsible for endangering human lives" is quite overdramatic. If people only pushed code to the kernel that they knew 100% was "completely free of bugs", we'd not have a Linux kernel at all. I've had to roll out enough new kernels due to security issues just this year to know that.

Yeah, on second thought, it probably sounds over-dramatic and all, but it would probably impose some valuable discipline for people who want to hack at the Linux kernel. (I've been programming for quite some time, but have never thought I am ready for going beyond userspace. Maybe I have it a little wrong, after all.)

I know you meant well, but to give such a stern warning and then admit that you haven't done any kernel-related work at all is frustrating. I've just been getting involved in kernel development for the past month, and it's pretty clear that the go-to taskss for novices is primarily writing device drivers or fixing bugs. I'm not sure what fraction of kernel developers get to work on the kernel core (CPU scheduling, boot/initialization, file systems, memory management), but I'm going to guess it's relatively small, or at least isolated compared to those working on drivers and other modules.

Further, if the maintainers were so cautious as to want to limit contributions to the kernel, you wouldn't see initiatives like the aforementioned (and awesome) Eudyptula Challenge, or Greg KH's "Linux Kernel In a Nutshell", which he chose to give away for free in order to garner more interest in kernel dev. The state of the project right now is clearly weighted towards getting new involvement as opposed to discouraging contribution because of extremely rare and unlikely cases like the one you mentioned.

I don't think you have to do kernel work to warn people they should be cautious when submitting to mainline. As a matter of fact, I've read the book you mentioned, alongside some others such as Understanding the Linux Kernel by Bovet and Cesati, and I've brushed against low-level kernel features such as memory-mapped I/O or epoll, so I'm at least aware of what might go wrong if you decide to write a kernel module just for fun and then issue a pull request against the original source tree.

I don't think contribution should to be discouraged, but I do think that people who have no sense of responsibility should be kept away from the kernel source. Kernel development is not supposed to be for everyone, it's hard. Great expertise and technical mastery is needed to write good, quality and responsible code that is suitable for inclusion into a kernel that today powers so many different devices.

I will admit, again, that my warning was probably too harsh, but I still stand firmly by the viewpoint that "casual" or careless kernel contribution should be a no-no. It's one thing to poke around on your dev system to find out how stuff works, but entirely another to write an official patch to a system that millions of people around the world rely on.

EDIT: And when I say that "kernel development is hard", I mean that it's harder in the general case than writing most userspace code. It requires you to think of stuff you don't usually worry about (think of the all the race conditions that might occur) and let go of all the abstractions provided to you by the kernel, it's harder to debug than userspace programs, and you have to support a wide variety of hardware platforms, each with its own quirks. In fact, I haven't done any kernel development just because I was intimidated by how hard it is. I've tried writing a kernel from scratch (OSDev wiki was a great help) but it soon came to be a buggy and unmaintainable set of kludges that was barely doing any real work.

EDIT #2: Why I can't reply to replies to this comment? Is there a conversation depth limit on HN?

"Kernel development is not supposed to be for everyone, it's hard."

Citation?

Given you've admitted to not working on kernel code, this seems like something you've determined out of reverence for the importance of the kernel, not something grounded in fact.

It's worth noting that the entire Linux project spawned from some "casual" development work by somebody doing it as a hobbyist project.

Your last point is just, ugh, oh so true. You don't get to be an expert in any field without diving head first into them at some point as an inquisitive amateur. Having a curious, naive spirit who isn't afraid to do things "the wrong way" and break a few things along the way is something we should be encouraging in our fellow hackers!

The OP's misaligned sense of responsibility honestly terrifies the child in me. I would not be where I am today were it not for the countless circuit boards I went through (nice way of saying completely destroyed) when I was 12.

True. If you follow LKML, you will notice that the majority of changes are to the device drivers code. And it makes sense, after all, you want the kernel to run on a wide variety of hardware. So, it's not just what fraction of developers work on the core but rather, what the fraction of changes belong to the core kernel - which is relatively small compared to the device drivers code.

Another good way of getting into kernel hacking is not just device driver books but following mailing lists you are interested in. netdev, pci-devel, kvm are some of the interesting ones. Also, checkout GregKH's driver project - these are drivers that are not yet fit of inclusion and hence reside in a "staging" area - a great way to get involved in kernel development ( http://linuxdriverproject.org/mediawiki/index.php/Main_Page). Note that some of the lists can have really high traffic but eventually you will get the hang of picking out the useful stuff. https://lwn.net/Kernel/Patches is a great resource for mention-worthy kernel patches without subscribing to individual lists.

I also agree with you that maintainers will _not_ be cautious and prevent newcomers. That's a bunch of bull. If at all, maintainers will be welcoming to changes from first timers and will thank you for your contributions if your changes do get in. Maintainers or other developers usually are frustrated with other style issues though - for example, lines greater than 80 chars, using a mail client that mangles your patch and so on. But keep in mind that the frustration is on the patch you posted, definitely not on you on a personal level :)

Hm. I'm a Linux kernel subsystem maintainer, and I think this stereotype ("only extremely accomplished programmers should work on the kernel") probably hurts more than it helps.

I'm not sure I've ever met someone who wanted to contribute to mainline and didn't understand the responsibility involved. What I think we see more of is excellent programmers who think they aren't good enough for the kernel, with the result that the average age for core kernel developers appears to be increasing all the time. It's not a great situation.

I'm not saying my opinion is representative of all kernel maintainers, but I'm personally in favor of losing the "you must be this tall to work on the kernel" rhetoric.

It's probably more to do with personality type than intelligence or anything else. It takes an extremely thick skin to handle the regular patch review slating dished out by most of the core team.. I simply don't want to deal with that shit for a hobby, and it'd take an impressive salary to make me deal with it professionally.

There's being opinionated, then there's being an ass, and I think there's far too much of the latter on the kernel lists.

If you look at the modern nerd, compared to the generation that birthed the kernel, they're more socially aware, fashionable animals that are more interested on working on sexy, visible things. Why spend your day dealing with bad attitudes and timing bugs when you can pick up $200k+ writing Ruby apps that enable sharing 5 second video clips?

I doubt the kernel will ever be starved for new talent, just that the ratio of people willing to work on it or deal with the social norms it entails is shrinking all the time.

I think a better quote would be:

"There's being opinionated, there's being an ass, then there's Linus Torvalds."

Don't worry though, he's not writing a three-paragraph rant calling you stupid, he's calling your idea stupid! Why do people take things so personally!?

...big ol' "/s"

Maybe the lkml is not always so bad, but I'd rather not deal with it and so far I have been content playing with my own projects. I feel kind of silly complaining about it when so many people manage it just fine, but then I see another explosion of aggression and drama and think "how are people okay with this?" I can't imagine having to work in an environment like that.

And then there's Theo de Raadt :D

>You could be responsible for endangering human lives.

I'm pretty sure what ever devices people depend on to keep them alive would be running a modified version of the Linux kernel with anything unnecessary stripped out and probably a lot of custom stuff.

They wouldn't just download it from kernel.org and throw it on there. I would hope not at least.

I agree. But I think kernel contributors should do their absolute best to ensure the code is as bug-free as possible. There are irresponsible sysadmins who'd use a Linux kernel where there is no place for a Linux kernel to be, and just because of that alone it would be a good idea to be paranoid when pushing to Linux mainline.

I wholeheartly agree with ❝devices people depend on to keep them alive would be running a -- version of the Linux kernel with anything unnecessary stripped out❞.

But the part with a ❝modified version (...) and probably a lot of custom stuff❞ made be shudder.

The Linux kernel is a very well maintained piece of software, and whenever, in the embedded world, I've seen modification or larger forks of the stock/mainline kernel, they always were of worse quality and had more bugs than what I (or everyone else) has gotten used to when running mainline code (as is customary on your normal x86 PCs).

That's because code-review actually works in Linux, and while it makes it sometimes very hard to get new features in, the ones that make it are usually well vettet.

Not everyone writes code for a critical Linux-based life support system. Very often it's just people who want to have a 7-segment LED display show the number of unread emails, or get a soundcard running, something along these lines.

And for them it's completely ok that there's a race-condition that might crash their machine if the kernel-module is unloaded just at the instant where the USB plug is removed -- that's something to learn, once the general grasp of how kernel-code with it's myriad of callbacks and hierarchies of subsystems has been reached.

Most crashes also will be catched by the Kernel and will leave you with nothing more than a

    [42568.139488] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
    [42568.139805] IP: [<ffffffffa003e008>] init_module+0x8/0x20 [my_dummy_module]
    [42568.140308] PGD 4450f067 PUD 4450e067 PMD 0
    [42568.140882] Oops: 0002 [#1] PREEMPT SMP
    (...)

in your dmesg, and maybe a dead "insmod" process in your process table. Make a note of the call trace, sync, reboot (maybe hard, using the magic sysrq key) and you are off for the next round of debugging. Try it, it's fun. Also there's kgdb which you can use when your bugs aren't that obvious.

I never had serious data loss (maybe the freshly compiled .o files will have 0-size, if I was in too much of a hurry to insmod ./my_module.ko. And even if, it will teach you to have an easily restorable development machine.

(I triggered the oops with...)

    int
    init_module(void)
    {
            int *ptr = 0x0000000000000004ULL;
            *ptr = 17;
            return 0;
    }

> Very often it's just people who want to have a 7-segment LED display show the number of unread emails, or get a soundcard running, something along these lines.

7-seg LEDs are best controlled by a microcontroller attached via USB or a USB-RS232 interface, as next to no one ships a parallel printer port. Soundcards these days usually all already have kernel drivers, I haven't heard of any modern chip without.

And USB stuff is best developed in userspace with libusb, it's way easier...

> it will crash the whole system

As anyone who's programmed in "completely open" environments like C64, ZX, DOS, etc. will be accustomed to, reaching for the reset button becomes almost second-nature after the first few times you mess up, and then you start developing a more thoughtful approach than just "compile and run, see if it works" when you realise that waiting for the reboot takes a significant amount of time. Maybe this is some sort of counterargument to the trend towards "early and often" testing cycles...

But I really think everyone should have attempted writing code for an environment with no memory protection and where bugs mean a reboot. It probably makes you a better programmer overall.

I agree with you completely. We shouldn't live in the past and fail to embrace the present, but I think everyone would benefit from knowing what the past was like. Maybe it will make you develop quite a few healthy habits, after all :)

Reply on news.ycombinator.com