I look at Ingo, Dave Miller, Richard Henderson and Andrew Tridgell who are ~ my generation, all both better coders than I will ever be and really nice people to work with.
Then I look at young coders like Olaoluwa Osuntokun and Bastien Teinturier who are also great to work with and who I can only keep up with because I have years of experience, and it completely keeps my massive ego in check!
Or are you asking how we create more people who will do this in their free time?
Also, as a Clojure dev, you cannot recompile the running kernel while using it at the same time using something akin to REPL driven development. That would be the other side of the live-patch productivity boost in so many areas.
If I am not mistaken, the CPU and other devices don't care, if everything is compiled ahead of time and a few functions tacked on later, as they are changed by e.g. patches, or if the kernel is completely recompiled. Obviously, I am simplifying a lot but it is a very serious question. We really should strive to build maintainable systems that don't have to reboot just to apply updates.
Here, you would need to involve a compiler of course and do more due diligence and then run ftrace or something to exchange the code. Also, you would probably in some cases need to make sure, the ordering of code is right as to not hurt performance. (See Emery Berger's talk https://www.youtube.com/watch?v=7g1Acy5eGbE which probably applies here as well.)
Of course, with great power comes great responsibility. You could add some optional checking facility, that would be able to roll back the changes to code if some conditions were met.
The rest could probably be solved by a very short "stop the world and switch a few pointers" routine.
It may well be that same sense of ownership creates an "inner circle" effect that is a barrier to entry for creating more such developers.
I've attempted something similar myself - a refactoring of a very dirty Java project that was orders of magnitude smaller than the Linux kernel. I didn't have good refactoring tools, nor even a good SCCS; my month-long effort had to be backed-out. I didn't enjoy it - it was boring, but needed doing, because the accumulating cruft was impacting my ability to work on the codebase (and presumably everyone else's). The project stayed dirty, and my reputation suffered.
Credit to Ingo for his perseverance. That is massive commitment.
It makes a big difference for the thousands of kernel developers [0]. There's also more than a million linux users globally[2], assuming only 1% might compile their own kernel, that still leaves tens of thousands of linux users benefiting.
Not to forget about the side benefits of a refactor. There could be runtime improvements coming out of this too down the road.
[1] https://en.wikipedia.org/wiki/Linux_kernel [2] https://www.zdnet.com/article/how-many-linux-users-are-there...
That said, in your case it felt a bit like an organizational failure also. As you could read in Ingo's mail he had to start over after plenty of work put in, for him there wasn't any reputation damage because he seems to have had free reign to start this as he wished.
Reading code is fun and it's a skill that took me time to develop. People need to told that's a skill you need and yes (shock) not emphasize creativity as much as we do. Creativity is great but it has enough encouragement in society already. We need people willing to read what already exists as well.
If you do a from-scratch rewrite, you could stay in igorance and might make those mistakes. But if you do a refactor, especially one broken up into many steps like this Linux one, it in fact demonstrates a better understanding the existing code and any nuance than merely leaving it in place!
Not touching scary-looking code is just bad, and not a wise form of humility.
But I think throwing away the changes should be an option if there's no objective improvement. Very often those changes end up on the main branch but they only cause changes, not improvements.
The real outcome is in someone's head.
That or they advocate doing a from-scratch rewrite.
I seem to get satisfaction from things that a lot of others might find tedious/boring.
People like us though just need to watch ourselves that what we’re doing is actually useful opposed to just scratching an itch. Which of course is not exclusive to us as developers often find many other ways of scratching itches without actually adding value.
On my best projects, I've done the first 90% so that a great many other can collective do the 2nd, 3rd, and 4th 90%s :). Without being "goal oriented" you can feel good about unlocking that part that would be too annoying for others, and they in turn can do the finishing work that wouldn't be fun for you.
Don't feel like you're not doing important work just because others are slotting in the keystones.
Good team builders will find a variety of different employee talents and interests and let people do what they like and develop those skills.
So we need to restructure our society to have economy the promotes leisure not output.
That will at least align the incentives right with a lot of more extreme engineering that just doesn't make economic sense today.
On a completely different level, yes, this is the sort of thing that does require passion. There is a lot of obscure stuff that is hard to plan or otherwise validate society. I'm a bit of a utopian I suppose in thinking if we had more leisure and guaranteed consumption, people would be able to do more passion projects, and we could give people hindsight recognition just instead like this.
Basic needs can be unconditional, renown can be the reward for "extra" work.
Finally, I hope as more monoliths are broken into libraries, we get more of this sort of stuff organically. Beyond the economics, conways law holds us back. (Even within the kernel!) Need to make sure people feel free and safe to really get down in other people's code, to develop the macro view, to see stuff like this.
This is the sort of "macro" refactor you can't get to with just local optimizations alone. We can and should break it into steps, but by not means does every commit have its own clear perf benefit.
In other words, in a perfect world, someone recognizes the value-add of your passion, and offers you money to do what you would do for free.
Evidence of this is all around us, and while most won't get this, some significant proportion of people pull it off.
So what do I win?
Health problems -> Money problems -> Fun problems.
You get the privilege to work on "fun problems" once you don't have "money" or "health" problems (otherwise, the "fun" problems are the least of your "problems").
Linus Torvalds worked on Linux for a long time before being directly compensated for it, but he probably would never have done so had he needed to worry about whether he could afford his next meal and rent that month.
I had free time, and an interest. Often the same characteristics of people who contribute to FOSS.
> The fact that they were still using mailing lists
In my personal experience, going back some 30 years or so online with my own personal paid email account – which is still live and still works – I find that most people I've worked or interacted with who dislike using email for workflow or other important comms do not know how to use email effectively. Strange as it may seem, this now applies to the developers of most email clients and online email services.
Email and mailing lists are remarkably powerful and capable tools, if used properly. Most people do not use it or them properly. I have yet to see anything else by anyone that is an improvement in every way on email.
> I'm not sure how it will progress when they retire or die off.
TBPH, as a Linux user and professional, I rather hope it does not.
Linux is an amazingly useful tool, but OTOH it's now, in and of itself, a vast hairball of technical debt. The traditional UNIX model itself is.
It is long past time that we should have moved past it on to better things. There have been many attempts but none have ever achieved critical mass.
At the rate things are going it looks quite likely that our technological civilization will collapse due to out-of-control global warming and the ongoing and accelerating mass extinction event. As the parent of a 2YO I very much hope I am wrong.
But if I am and it doesn't, the world needs better tech, and a ½ century old UNIX clone already does not really cut it today.
If Linux and all the other monolithic Unixes die when their dev teams die, we'll have to move on to newer, smaller, simpler systems that a new generation of programmers can actually read from top to bottom, understand, and do useful work on.
If monolithic Unices become the mid-21st-century COBOL, just kept around for a few old systems and only critical bugs patched, that will be a big win for us all.
Linux, perhaps, but "the model"? I'm not so sure.
> It is long past time that we should have moved past it on to better things.
That's what they said about SQL RDBMSes too.
> There have been many attempts but none have ever achieved critical mass.
Maybe for good reason.
> the world needs better tech, and a ½ century old UNIX clone already does not really cut it today.
That's what they... Eh, I'm repeating myself.
> If Linux and all the other monolithic Unixes die when their dev teams die, we'll have to move on to newer, smaller, simpler systems that a new generation of programmers can actually read from top to bottom, understand, and do useful work on.
That's probably what they... No, not quite. But it may have been what Linus thought -- at least, judging from what he made.
> If monolithic Unices become the mid-21st-century COBOL, just kept around for a few old systems and only critical bugs patched, that will be a big win for us all.
I doubt that'll happen. I'll be happy if they become the mid-21st-century SQL -- which I suppose will also be still going strong.
But the alternatives are there.
The fast-headers thing put me in mind of the way that Plan 9 changed the C language, not only formalising indentation and things, but to forbid nested #includes, which apparently vastly reduced compilation times.
Inferno moved on to largely supplant C with Limbo, which is one of the ancestors of Go. I'd say Go, D and Rust all show that there's demand for a newer, better C and that C++ is not it.
Apple's xnu is widely held not to be a true microkernel because of its big in-kernel Unix server, but Minix 3 is... and for all that Minix 3 is not quite there yet, QNX shows this is possible and doable and performant.
It's 2022. We have boxes with terabytes of RAM now and hardware-assisted virtualisation even on £5 ARM boards. We don't need a total clean-sweep replacement; we can virtualise the old stuff and slot in underneath something smaller, simpler and cleaner that mere mortals can understand in less than a lifetime.
3D Xpoint memory is on the market now and makes non-volatile RAM doable, multiple orders of magnitude faster and longer-lasting than flash memory. Computers don't even need drives any more: we could just have a few terabytes of persistent memory, and no more shuffling stuff into memory. True single-level store is doable now. Who needs files when everything is in RAM forever?
ISTM that much of the computer industry has its eyes on the ground now, and is just shuffling round and round in the same old groove, rather than trying new stuff. In the middle of my working life there were dozens of exciting new OSes trying to do things differently. Now, we have 30 million lines of C instead, and need a billion-dollar industry to aim all those many eyes at all those many bugs.
As Molnár himself said: “Don't forget that Linux became only possible because 20 years of OS research was carefully studied, analyzed, discussed and thrown away."
Very few companies or other management structures would ever sanction this kind of work. They might recognize it as important, but it's too big and open ended to "allocate resources to". So for this to happen you need to pay people for more of a general "make stuff better for us" role. Even that's hard because you never know what you're going to get.
Without them, quality degrades and you get a lot of busy work from working in inefficient systems.
I did this at work too early on in my career too but it was quickly made clear to me that if there's no ticket that the customer has opened (or at least approved), they're not paying for it and I can't spend time on it. Tickets I opened myself were 99% ignored or only brought up when a problem I had anticipated actually manifested in product (told ya.. now this 18-month-old ticket is suddenly relevant?). I quickly learned not to give a crap about the code base (hard to give a shit if you're not really allowed to give a shit?). I guess that also slashed my motivation (and productivity with it). It's pretty frustrating to work this way.
Creative freedom and autonomy are key. I'm sure there was no micromanager assigning Ingo Molnar the "make kernel builds faster (est. 80 hours of work)" ticket.
Obviously this is an extreme case, but it scales. Something like 8 years ago I noticed something was horribly wrong with the ARM32 boot wrapper, submitted a fix, and my approach was mostly shot down by the maintainer. Last week I submitted a 34-patch RFC series to finally add WiFi support for 5 years' worth of Macs (which required quite a bit of new scaffolding in that driver, as well as fixes and new features) and it's gotten positive reviews so far, modulo nits. I wouldn't have been able to pull that off 8 years ago. And it's not like I spent those 8 years doing kernel dev, but I've sent in a few fixes and watched how other kernel developers work. I started on a big kernel project a year ago and all that sitting and watching has been very helpful in putting out stuff that people like.
1] remove their technical blockers. Often a person like this doesn't have a universal skillset — they're usually good, but they can't do everything. If there's a 5% of the job that's hard-blocking them, make sure it gets done so their tires don't get stuck in the mud.
2] remove their bureaucratic blockers. Make absolutely sure the project WILL proceed, even in a skunkworks capacity (this is the primary value of skunkworks — it's the ability to proceed with something you know will work, in spite of authorities expressly forbidding it because they think it won't work and don't want it to happen for what's usually a petty reason like a "it's waste of resources").
3] remove their emotional blockers. This is really the apex — THE primary value of a person like this is not technical skill, but rather, their work ethic. A person like this has a really profound, train-engine drive to just keep soldiering on. The danger here is the problem of the "weary crusader". This willpower is considerably above average ... but it's not infallible. It's not like a superhero that's always gonna come through no matter the odds. They can break down. They can lose heart.
One of the things that will constantly break them down is if people are naysaying them, and calling into question the value of their work. If they're doing a giant refactor, and people are angrily opposing it simply out of fear of change, it will break them down. As silly as it sounds, you want to coddle them — they might be unusually emotionally tough, but they are the absolute last person you want breaking down. Treat them like a "snowflake".
And yeah, I'm explicitly suggesting censorship. Don't permit negativity. Don't permit the usual cesspool OSS discussions where you have a bunch of people who contribute almost nothing bikeshedding a project to death and exercising a sort of "liberum veto" on any attempt to do new things. Just shut down those conversations — make it clear that only the people who do major heavy lifting (or really, who are discussing things "in good faith") have a voice. The whole OSS community has an unhealthy cultural fixation on "absolute free speech", and like ... I completely understand where that comes from, but it's absolutely toxic for motivation. I've observed over the years that a lot of the more aggressively successful groups at "creating things" basically just establish their own safe-spaces where people get emotionally reinforced instead of emotionally sabotaged. Occasionally their work will get exposed to the world and some toxicity from outside will leak in, but the bread-and-butter day to day experience of working on their projects has them surrounded by friendly people who believe in what they're trying to do, and encourage them to keep going.
Partly because we know we won't be there forever to keep working on it. If we feel like we're part of a group that "has our back", and will support what we built, and grow it into something, it's a million times easier to keep working on something, compared to a situation in which someone actively holds our work in spite and is itching for the first chance to tear it down the moment we turn our back, or move on, or die, etc. This is all "emotional extrapolation", but these are the sort of depressive/anti-depressive thought cycles that go through your head during a mega project, and too much of the negative side can just break people.
Frankly everything I know about software dev is geared to discourage exactly this sort of behavior.
I would not be happy to see a pull request touching the entirety of the code base for better compilation performance. Not because such a thing wouldn't pay dividends in developer performance or code readability, but because such a change is just inherently risky.
You might be able to train more people with whole program understanding (Definitely a skill that is lacking, IMO). But I doubt you could train someone to be able to "touch every file in the kernel and get it merged".
Most highschool teachers are really bad at this.
Also, you need to have gathered the skills to do what you wanna do. But you'll put in the effort yourself if you're inspired.
That'd be a challenge. He is obviously a very talented person who grew up behind the Iron Curtain and went to university just after that collapsed. The resource constraints bred creativity, a certain kind of hacker mentality. The lead developer of Crysis: Warhead and Crysis 2 (and indeed most of the former Crytek Budapest team) is another extremely talented programmer emerging from the same time. Palma sub pondere crescit!