“Because I'm a bastard, and proud of it” (2000)
lwn.net
lwn.net
Firm but fair kindness, diplomacy and a lead-by-example approach are the keys to leading a technical development effort over the long term. Do make tough decisions, do hold people to account, do work hard, do expect the best from other people, do get rid of problem team members and do address under-performers, but all that can still be done with patience, kindness and diplomacy.
I made the mistake in my early career years of modelling my leadership style on Bill Gates also known for the same abrasive, no holds barred style. It doesn't work.
Both work. No seriously both work. It's just politically incorrect to run Dad-style projects or companies now.
It's kind of ironic to see that, nowadays, with the increased necessity for concurrent and multi-agent systems, a debugger becomes even more obsolete and cumbersome, since you're often fighting a race-condition or some kind of resource starvation.
Personally, I see his rhetoric in this email as highly intelligent community leadership in which Linus is arguing for better systems design (inside, or outside of the Linux kernel).
Solaris has a kernel debugger and a more concise architecture.
Personally, I think the UNIX OS design is a poor fit for the requirements of Docker et al. The problems with cgroups should be a signal that it breaks the fundamental design assumptions of Linux (doing fundamental resource separation and multi-ring security in a monolithic kernel).
I'm struck by the analogy that I entered the programming game juuuust after front panels went away from minis, a long time ago, and there was much crying about how the programming world really needs front panels to debug code at that time. Nobody demands a front panel to debug code now, of course.
Anyway my point is I wonder if something better than a debugger will ever evolve. Much like front panels were replaced by technology or were no longer relevant for the task. A smart AI agent providing advice? Crowd sourced code reviews? Maybe nothing at all, unfortunately.
Another thing to realise is: a debugger works fundamentally on the sequential composition operator (putting instructions or statements after each other). We have to realise that the composition operator is very complex, hard to reason about and difficult to abstract (it's why modern build tools use a dependency graph and no shell scripts). As such, we should prefer other compositions in order to have correctness flow from our design, instead of our debugging cycles.
Both Linus and Bill Gates were extremely successful. Are you claiming they could be more successful with a softer style?
I have another conundrum. Assume both these individuals would have to to do some real amount of work to change their personal style. This translates to time spent not doing the other things they are perhaps excellent at.
Wrapping your argument in nice words all the time is actually really hard. It takes time. Frankly it is not always necessary depending on the relationships between individuals.
Let's say you and I start an open source project and we both find it really hard to say things nicely, but we have X hours a day to contribute and get stuff done. So I propose to you: "Hey, let's get in the habit of saying what we feel. We'll remember and accept that humans aren't always rational and we're going to hurt each others feelings sometimes. We'll remember that we all say and do dumb things all the time, so it's no big deal to call somebody out for it. But we think this will save us time in the long run, since we suck at sugar coating our words and feelings."
What stops us from building a highly successful project of like-minded individuals? What do we lose when somebody comes along and says "hey, you have to be nice now!"? Should we refuse to work with such people?
These are real questions and I don't think they have clear answers.
There's no doubt that better tools have let us be more productive, but in the process we understand the code we write less deeply than we otherwise would, because we are required to spend less time contemplating it.
One of the insidious things about supermarkets is that they let you be lazy when getting food. When you don't have a supermarket, you need to be more careful about what you do and really understand your food, because a misstep when collecting food can take lead to death. With a bunch of safety nets at hand, you feel like you can just blow out and get some food and work it out via diets later.
There's no doubt that better tools have let us be more productive, but in the process we understand the food we eat less deeply than we otherwise would, because we are required to spend less time contemplating it.
It's absolutely true what's being stated here. Yet when you apply it to anything else it shows it's utter stupidity. Without tools we are less, it's as simple as that. Doesn't particularly matter what tool you're talking about, debuggers or supermarkets or cars.
Let me tell you the one advantage that producing throwaway "software" that is 3 page webpages, at most, with 2-3 features. The one property this has is that it's cheaper. Due to a horrific accident of history it has an easy and mostly idiot-proof distribution system it also pays more.
I would also like to point out that when desktop apps were the norm, and an environment without debuggers just simply didn't exist, desktop software was large bodies of code where you could fill books with their feature list. In fact people did [1]. Today a webpage with 5 screens is considered a lot. Not doing massive code reuse was considered lunacy, where now nobody does it.
But let's not kid ourselves. Software 10 years ago beat the crap out of webpages and android apps in features, documentation, ... everything.
[1] http://www.amazon.com/CorelDRAW-X6-The-Official-Guide/dp/007...
And I say this working in an area where debuggers either kind of suck or are limited (graphics programming), so I have plenty of experience working on complex systems with no debugger. Believe me, I do this exact painstaking thinking when I write a shader (no break points or even print statements), and yet the shader code I write is about the same quality as any other code I write, it just takes longer.
We'd have less, but much better software.
For an opposing example, look at the world of JavaScript.
But we should abstract away from the language. We've had 35+ years of experience of designing and implementing user interfaces, including their design patterns (MVC, Action/Event based). History is filled with successful designs.
Just consider a sophisticated tool such as your standard IDE: do/undo functionality, separation of concerns, event based, MVC, good standard graphical toolkit, easy deployment and upgrades. These designs are not new, and date back to at least the late 90's (Delphi).
What changed? Well, we've been doing incremental design: throw something at the wall, see what sticks and debug the hell out of it while adding new features. Perhaps that design philosophy is not very tenable in the long run.
That's hardly what I'd rate as "primitive".
That a lot of people carelessly throw together CRUD apps does not contradict what I'm saying. In fact, I'd say most CRUD app development is highly primitive, notwithstanding the complexity of the web platform itself.
Debuggers do tend to sometimes to mask heisenbugs, but this goes the same for just about any tool performing intricate binary analysis. And there's some subtle timing bugs where it's a fact of life that you'll get misleading readings.
Fact is, without debuggers, most highly subtle war story bugs would go unsolved. Logic analyzers, tracers and ICEs are debuggers, too. "Debugger" doesn't just mean gdb. Not that there's anything wrong with it.
There's a ton of programmers who can't code without a debugger because they don't really understand what's going on any other way than line by line. Those people would be filtered out. Debugging doesn't require a debugger, it's just the easy way to do it.
The key to better software is learning from the research literature, not thinking that you're improving because you write nothing but ASM macros in ed(1) in a TTY emulating a VT100 from the 70s.
Sure there, it's been said multiple times; those still writing software would be the smartest around. That alone is reason enough to think the overall quality would be higher; if you don't believe that, then imho you don't know what overall (average) means. You're not listening, you're focusing on outliers and I'm taking about the mean. It's a valid logical inference that if you weed out less intelligent developers by making developing harder, that the average quality goes up; not the absolute quality, the average, mostly because you'd be weeding out tons of bad developers who write tons of bad software. This really isn't debatable, it's logic. Weeding out the bad raises the average quality, there's nothing to argue here, that's just a fact.
Theo de Raadt, on the other hand, has never stopped hacking since his days in NetBSD.
Ordinarily, when someone is denying an offer or rejecting an idea, they're going to put on the kid gloves and try to make sure you feel good even though they're doing something that, for all intents and purposes, shouldn't make you feel good. Although there is no negative intent when someone is attempting a "sensitive" approach, it can be interpreted as a form of dishonesty and condescension: "why is this person pretending to be doing something good for me when they're so obviously not? do they think I'm so dumb I won't notice?".
Frequent contact with that communication style sometimes triggers a deep-seated resentment for it. By the same token, some people are deeply appreciative of the general concern for the recipient's emotional health and self-worth reflected in such statements and are taken aback when they come in contact with someone who doesn't choose to communicate in that way.
It's hard to find a middle ground that works for everyone, regardless of their background or perspective. I guess that's why we have professionals dedicated to finding the ideal phraseology for corporate communications. Maybe it could be argued that we shouldn't we try to do that.
It seems to me that these two things are not actually logically connected and we can have both: underthought code is a unrelated problem to a debugger.
Its like saying you can't build world class woodwork if you have a router. Instead you must only have awls so you have plenty of time to think about it.
It strikes me that simply means there will be less software in the world at much higher dev cost and that isn't a good thing necessarily.
I think I understand his point though. Kernel debuggers aren't amazing by any means, not at all comparable to standard application debuggers, and a lot of the actual hard problems you run into when dealing with a kernel can't be easily solved by debuggers anyway. Single-stepping through kernel code is filled with bugs and he's right when saying you can't really trust it. Kernel debuggers can be helpful, there's no denying that, but if you're relying on a kernel debugger to solve your problems then you're doing it wrong, and I think that's his point.
There's too much that could go wrong that a kernel debugger couldn't easily catch for you - And I'm sure the debuggers in 2000 weren't as good as the ones we have now. Not having a debugger removes the risk of people doing that, and he'd rather have less kernel developers that are more aware of the code they're writing, then more that don't know as much about it.
Gentlemen, I am well aware that many woodworking shops have a FooBar2000 scrap wood burner. This shop will not. I am not saying woodburners are always bad, or the FooBar2000 is individually bad. Or pretending that scrap never happens. Why I enjoy a warm toasty winter night in front of a fire like the next lad. However having a scrap wood burner implies we create scrap that needs burning. And I will not tolerate that in my shop, nor encourage other shop owners to recklessly generate scrap.
Gentlemen, here we make keels of ships. The entire ship relies on this keel never breaking, creaking, cracking, groaning. Upon those ships... Lives depend. Businesses depend. Governments depend. And I will not tolerate scrap as a boat keel and all the destruction failure can cause. Therefore given that we can't afford to create scrap, we will work slowly and methodically and intelligently so as to not create scrap, and thus have nothing to burn in the FooBar2000 scrap wood burner, thus we do not need one. In the unlikely event that I or someone else make a keel that needed some scrap to be disposed of, well, if it tests good, thats OK. But I'd never tell a dude setting up his own shop that if he's going to build boat keels he badly needs a FooBar 2000 scrap wood burner because he'll probably screw up a lot, because if he's screwing up a lot, I don't want the world depending on his boat keels and I won't accept that caliber of workmanship in my shops nor stamp my name on it.
Gentlemen, you may not like my opinion about your deeply held beliefs about woodworking. I sympathize. I don't agree with everyone else either. Thus I'll give you a psychological outlet by talking like a pirate for a bit now, and making light of your mother's parentage and something about elderberries. Thus if you're insulted by my technical evaluation of your woodworking skills, you can tell everyone else you're actually insulted by my talking like a pirate. Then everyone goes home happy, and the world gets safe strong indestructible boat keels.
https://www.freebsd.org/doc/en/books/developers-handbook/ker...
And better code than Linux.
It's a matter of opinion whether their products have suffered, now that neither company has a Steve or a Bill or a Linus running the show. My opinion is that they have.
If he were to actually speak plainly, 50% of his email would disappear as overly repetitious.
History suggests otherwise.
Full disclosure: I don't use debuggers myself, partly because it isn't practical (I spend most of my time in Go these days, rather unwillingly I might add, and Clojure. Both languages aren't terribly friendly to step-debugging.) but when I need one I really, really need one.
Which religion would that be?
(maybe call it "Pseudoscientism" a belief that merely scientific-looking commonalities are in fact universalities)
Can you clarify what you mean with "this kind of thinking"?
If you can agree that religion is merely institutionalized ideology, I think by contrast you could look to Aristotle, whose lifespan intersected with Plato's, who took the opposing view: Describing what he has observed about bees, he writes "the facts have not been sufficiently ascertained, then we must trust perception rather than theories."
The Greeks formally established the mechanics of inductive reasoning as the formulation of truths from empirical evidence. Plato, for all his gifts, stands firmly in contrast to this. Ideology is fundamentally about turning away from evidence and making the world fit into your polemic.
If you can agree that religion is merely institutionalized ideology
I do not fully agree, as not all ideology would qualify as religion, even if institutionalized. For example, capitalism is not a religion yet it is both an ideology and thoroughly institutionalized. I would agree that religion is a form of institutionalized ideology, but not merely.
But I (think I) understand what you're saying. You're saying that because Plato considered fundamental truths unknowable, he made his vision of the world subject to dogma concerning the interplay between the unknowable and our reality. That is certainly true, but I'm unsure how much prescriptive dogma is in Plato's works.
Regardless how his theories have been adapted by Christianity later, for as far as I know Plato himself:
- never declared empiricism as futile,
- did not give his "Shapes" any agency in our reality,
- did not create his own narrative of good vs evil.
In fact, Plato was quite direct in his approach to morality: people should do "good", because doing good things would bring happiness, and doing bad things would create unhappiness. That is a quite utilitarian approach to civilization, and I would not call this a religious stance. Plato did consider the soul to be immortal, but he also said that learning would begin anew in the next life, so I don't think he subscribed to any form of afterlife judgement.
Consider this tradition survives even in Newton, who was an avid alchemist, wrt The Philosopher's Stone and obsessed with the numerology of the dimensions of Solomons Temple even as he worked on his Principia.
I feel kinda incompetent as I am missing something here, does the inclusion of a debugger, even when not in use hamper performance or anything in particular in anyway?
Is it the same type of debugger, where one could step through code and inspect objects or is this something completely different when it's related to the kernel?
Linus, in his own way, is pointing out that an ultrasound will not identify that your house is missing a wall.
Linux is vulgar, but that's pretty much it. Doesn't mean he's a bad person. His opinion can be bad without resorting to bashing his personality style.
P.S. Ask yourself why you have double standards with regards to "bashing".
Exceptional.
I refrained from guessing the cause of this before, but I believe it is you being a fan of Torvalds' and/or being sympathetic to his unwilingness to moderate own behavior.
Please address my point about definitions or explicitly introduce a new point if you really want to argue the other thing further.
Be the change you want to see. Going off topic to bash his personality style doesn't really add anything to the discussion of Kernel debuggers.
You're right with me being sympathetic, I don't hide that but it's not relevant to my point. Thanks for being curteous.
The double standard is how we attack Torvalds for the same thing radical feminists say on a daily basis. You ever interacted with Shanley?
I don't care about their personality style. They're both abrasive, I just think Torvalds has actual value and skill sets that I could only dream of. The other one... Not so much.
Going "OT" was due to the title. I thought it was that post where Torvalds obstinately defends his rudeness. Basically the man's lack of style almost prevented his point from being communicated. So even digging for points which I did and do* only went so far.
*Mind you, I've found sifting through noise to have diminishing returns. People often use cuss words instead of arguments. Strong points do not need strong language.
What nonsense. You can apply his exact same logic about kernel debuggers to reject all advances in software made after 1968. If only all programmers used nothing but DOS and ASM on 4-bit CPUs, we would finally be liberated. Finally let "Darwinism in software development" take its course.
(Of course, I'm sure most people commenting here have never hacked on anything more sophisticated than a V7 Unix derivative.)
Linus is an uncommonly gifted low-level x86 hacker. But his technical judgment is mediocre. I just don't get why he's held in such esteem.
Aaah, the myth of the Protestant Work Ethic.
Look we're all devs here, and everyone who's written more than fizzbuzz has run into judgment issues about speed. A military campaign analogy fits really well. You can advance slowly, defensively, without possibility of failure or accident, with full knowledge of every little step you take, every footfall of every process is understood and safe. In which case your code will never crash and people can run nuclear reactors with it and coincidentally a debugger would be a complete waste of time because it'll never be needed. Or you can ambitiously run ahead in a reckless offensive charge, probably right into a swamp or ambush or minefield, and then you'll need a debugger to pick and prod your way out of a self caused disaster. That obviously requires a debugger. It also results in code that possibly thru sheer luck didn't crash into the debugger being shipped, and then killing people when the nuclear reactor controller crashes or whatever. There are whole swaths of logical errors and race conditions and spec problems that debuggers won't help with but sitting there and thinking hard will avoid, and nobody wants to find all those mistakes the hard way in shipped deployed code, and then Linus has to find what should have been an obvious bug had the dev slowed down and thought a bit first, etc.
If you needed a debugger because you were too lazy to check a memory reference before you freed it or whatever, that's probably the quality level of code that also "forgot" to check for trivial buffer overflows or null termination of strings or just basic foolishness. He just doesn't want it.
Note that this is ultra low level kernel code that entire systems are based on. You have to be more paranoid and slow and methodical than everyone else with this kind of code. Probably the completely wrong attitude for someone who just installs wordpress, etc.
Note that he expressly denies the work ethic... if you can send him production quality code despite having used a debugger and despite having intentionally avoided thinking hard, if its none the less good he explicitly says he will take it... he (and I) just think it very unlikely that situation would occur. If he demanded the work ethic he'd be insisting on testimony that you meditated on that function for a minimum of two minutes per line or something equally effort based rather than achievement based.