Comment by evmar | Hacker News Reader

Comment by evmar | Hacker News Reader

Linus wrote: 'Even a fully built kernel ("allmodconfig", so a pretty full build) takes about half a minute on my normal desktop to say "I'm done, that pull changed nothing I could compile".'

Since it's just after a git operation all the file state should be warm in the disk cache. It really shouldn't take that long. The Linux kernel (at least by looking at [1]) is about the same size in terms of file count as Chromium, and we got this operation down to about a second by using better tools (i.e. non-recursive make and then eventually a replacement).

I appreciate that the kernel has its own requirements (it sounds like his no-op builds are still running shell scripts, something you ought to avoid in your critical path) and also it's great that he's running it this way in part to help profile a "normal" workload... but I'm also a bit sad to see so much time spent waiting for something slower than necessary, as well as time spent optimizing what feels like the wrong thing.

[1]: http://larjona.wordpress.com/2011/06/15/numbers-about-the-li...

From further down in the comment thread, Linus says that speeding this up should help other workloads too, and he's sick of using make replacements:

---

+Peter oh, it's absolutely true that 'make' is a pig, and does too much, and we don't exactly help the situation by using tons of GNU make features and complex variables and various random shell escapes etc etc.

So there's no question that some "makefile compiler" could optimize this all. But quite frankly, I've had my fill of random make replacements. imake, cmake, qmake, they all solve some problem, and they all have their own quirks and idiocies.

So while I'd love for 'make' to be super-efficient, at the same time I'd much rather optimize the kernel to do what make needs really well, and have CPU's that don't take too long either.

Because let's face it, even if the kernel build process was some super-efficient thing, real life isn't that anyway. I guarantee you that the "tons of small scripts etc" that the kernel build does is a real load somewhere totally unrelated. Optimizing page faults will help other loads.

Maybe he'll get so fed up with make that he'll write his own that everyone can just switch to instead of all the crud we have.

Oh no. Not on your life. It would be better for the entire world if Linus had a heart attack rather than rewriting Make. Make is already unusable; git-style make would be an abomination.

Can you even imagine reading the man pages of all the different git-make commands? The syntax?

I hope Linus keeps contributing to humanity and doing what he's doing. He's obviously a great engineer, and if he works hard, he might undo the wrong of releasing git into the world.

Huh? Since the comparison would be to the likes of svn and cvs, git is a hella good improvement.

If you are willing to learn what it is that it does, git is actually fairly frigg'n good.

Of course, mayhap I've just not used a better tool.

> he might undo the wrong of releasing git into the world

I've never seen a comment with which I more vehemently disagree.

I can't tell if you're joking or not. The popularity of GitHub and its position as the defacto modern source control management tool demonstrate that very few people hold your point of view. Sure, Git has a lot of options, but what do you need to actually be productive with Git? Probably only a handful of commands: git clone, git pull, git push, git commit, git add, git checkout, git branch, git log. And not too much more than that. At my university Git is used in every single lab based CS course and I don't know of a single person who has had trouble learning to use it.

One of the main criticisms of git has been its user interface (which to be fair, has improved considerably over time). People often compare it to Mercurial's, unfavourably.

No, it demonstrates that things that the Linux kernel adopts tend to get adopted. That isn't very surprising. Git being faster than the dVCS's of the time probably helped. Popularity doesn't equate to quality. QWERTY is still the default keyboard layout.

I am willing to bet money (let's say 0.1 bitcoins) that there are plenty of people at your university who have trouble with Git. We can haggle on terms, but I'd bet that you'd see 10 people at least struggling with git in an intro course.

Once you learn to give it the right incantations, and learn never to deviate from the path you know, any tool can become usable. But it will never be a part of you and will never make you stronger.

This has crippled a generation of developers and I'm afraid it will be a massive barrier to entry, stunting humanity's growth probably by 10-20 years.

See http://homes.cs.washington.edu/~asampson/blog/git.html for a more in-depth treatment of Git's insanity.

QWERTY exists because alternatives were not objectively better. Everytime anyone proposed an alternative, they had to not just be a little better, they had to be so much better that a lifetime of QWERTY muscle-memory being undone was justified.

It's the same reason so many new display technologies have failed: its not good enough to be better then an LCD eventually. You need to be better now, and better then the LCD which will come about as evolutionary improvements in manufacturing too. Being a little cheaper in the initial plant cost is meaningless if the plant has been built and we understand its processes.

Which of course is why the commentary on Git is absurd: people struggle with Git? People struggled with CVS. People struggle with the notion of "files" and programming in general. The alternative has to both exist, and be easier to use from the get-go. Not just a different set of traps.

Relax. Deep breath.

At some level, command-line conventions are arbitrary. "ls" could just as easily be "dir" or "list." "revert" could just as easily be "checkout" in some contexts.

UNIX has never been a zero-learning-curve OS. That's OK. There are other operating systems that fill that role. git is a UNIX tool which is intuitive once you learn it.

The world doesn't owe you anything, and if you don't want to learn git, or any aspect of programming, nobody is forcing you to. There are lots of other things to do out there. Some of them even pay more.

I downvoted you because of your ridiculous hyperbole.

Hah ha,

The parent quote doesn't deserve to be downvoted but it seems to have been massively downvoted for the same reason a massive number of people use GIT who shouldn't - the bandwagon effect.

Git may well be a perfect tool for the Linux kernel but it's still an opaque nightmare for the many average non-kernel-developers who use it as "the new good version control system" (which it isn't, it's only a tool for a specific purpose made by a famous person).

The poster's point is valid - if Linus created a make replacement for his purposes, the effects of everyone else inappropriately adopting it would be horrific (even if it was indeed, a great make for the kernel and just for the kernel).

> Because let's face it, even if the kernel build process was some super-efficient thing, real life isn't that anyway. I guarantee you that the "tons of small scripts etc" that the kernel build does is a real load somewhere totally unrelated. Optimizing page faults will help other loads.

This is true as far as it goes, but a silly argument. Let's apply a reversal test (http://www.nickbostrom.com/ethics/statusquo.pdf‎).

Suppose the kernel build were as efficient as Chrome's is claimed to be on this page (<5s) and wasn't stressing his system. Would Linus then approve of anyone submitting patches to deliberately slow down the Linux kernel build just to show up slownesses in page fault and encourage kernel devs to spend time on optimizing that?

No, of course not! That would be idiocy and the person submitting the patches would probably be banned by Linus in his titanic rage. However, since the slow build & page faults is the status quo, Linus is making lemonade of it...

Not sure why, but your URL has a bunch of junk on the end of it.

I don't get what you're arguing against. Even if the kernel were superfast, there's probably another "real load" out there somewhere that legitimately runs into lots of page faults.

It's a question of what should be addressed first.

Linus is actively arguing against a faster kernel build. It sounds like it makes sense, because it gives him leverage with CPU vendors.

But it doesn't, really. If kernel build were already fast, he would never in a million years slow it down just to get that leverage.

Now maybe he's perfectly aware of this status quo bias, and he's taking advantage of it to meliorate something he would otherwise have no power over. Sneaky.

Still, what's the priority? He's made his point now, hasn't he? He could work on making a faster build process, now.

Slowing it down wouldn't give him the same leverage that it has as a real use case.

Anyway he's not saying slowness is better in and of itself, just that there are better places to work on than replacing make.

Actually, his main point is "here is a workload that is kernel-bound that ordinary users deal with, and here is where the time goes." He has explained in the past that he works on whatever interests him, period. So profiling page faults must be more interesting to him than speeding up the build per se. That's all.

I think you missed the point. He's not denying that make is slow and could be faster or that most of it doesn't have to do with page-faults. All he's saying is that page-faults in general are slower and speeding them up would improve lots of various different kinds of loads besides just the kernel build system (Which would probably not improve very much from just that alone).

I'm sure he'd be happy to take patches that make 'make' faster, but that's simply not what he's trying to address here.

What?

"When a proposal to change a certain parameter is thought to have bad overall consequences, consider a change to the same parameter in the opposite direction. If this is also thought to have bad overall consequences..."

How is speeding up kernel build times a "bad overall consequence"? Thats exactly what Linus is trying to do.

Except Linus is not addressing that problem directly. If he was, he would probably be writing a make he'd be satisfied with. Instead, he's working on page faults.

Speeding up page faults does _directly_ improve build times. I don't understand why you think writing yet another make is the obvious direct fix, but fixing page faults is not. Or why you think Linus shouldn't fix things he is clearly interested in.

So, that is actually an interesting thought experiment, thanks for that.

However I'm not sure it is directly applicable here. There are two courses of action that could solve this problem:

A, an action that improves kernel builds

B, an action that improves several workloads

For A and B of similar cost, it makes sense to do action B in preference to action A.

Your argument speaks to A being of positive utility, but given a finite knapsack of effort smaller than the set of possible actions that fit in the knapsack, a greedy algorithm that places any positive item into the knapsack is not optimal.

I don't follow your reasoning here, but let me expand my observation further: Linus can A. improve his build system (low-hanging fruit which the Chrome numbers suggest could yield an order of magnitude better performance), or B. he can search among a variety of difficult unlikely-to-yield-major improvements (like yelling at Intel engineers 'make it go faster!') which will improve his build system and also other hypothetical loads (which are unlikely to be large gains if any at all; what, is Intel too ignorant to try to make the TLB fast?). He is claiming B is better in part because of the hypothetical loads makes it better in total.

Most people would consider A a more reasonable reaction, especially after hearing that Linus's best idea for doing B is apparently going all the way down to the hardware level in search of some improvement. We can see this by intuitively asking what people's reactions would be to a proposal to induce B if the equilibrium were already at A.

On the other hand, Linus is one of a handful of people in the world who may be in a position to get results by yelling at Intel engineers to 'make it go faster!'. This isn't because Intel is too ignorant to do things on their own, but because practically everything can be optimized further, and Linus may have enough sway to focus the engineers' attention on the problem that he wants solved.

Personally, I don't care much about the speed of the Linux kernel build system, but I do care about the speed with which page faults are handled by the CPU. Even if the chances of success are lower, if he is able to succeed in speeding up every page fault on future Intel processors, I would consider that a much greater good.

The real problem (as I see it) is that I think he's trying to optimize the wrong thing. His worst-case test is based on trying to repeatedly fault in an uncacheable page: every lookup TLB lookup fails at every level of the cache. Likely, Intel has chosen to optimize the real situation where page translations are cached when they are repeatedly accessed.

Most people would consider A a more reasonable reaction

Improve your dev cycle inefficiency that you don't really mind, or improve your product? What's unreasonable about the latter?

especially after hearing that Linus's best idea for doing B is apparently going all the way down to the hardware level

He's building a kernel, not a webapp. "All the way down" is a single layer.

Intuition is probably not a good guide here. An improvement to the kernel build process is relevant to the thousands of machines that are used for kernel development; an improvement to the page fault speed is relevant to the over 1 billion devices that run the kernel. That's a pretty big multiplier.

(I believe the GPs reasoning is that "improve the build system" and "retard the build system" are not symmetrical, because both directions require positive effort to be expended).

On the other hand, if build system improvements truly are low-hanging fruit, then there are a lot of people out there who are capable of helping. A much smaller number of people are capable of productively working on page fault optimizations, so having someone like Linus address low hanging build system fruit instead would be a waste of talent.

We have a different prior expectation of the utility and feasibility of action B. In my view, Linus is uniquely suited to do something useful here, and his chance of success is high. In your view, Intel is already doing the best it can, and his likelihood of success is low. Our positions follow.

A isn't really very low-hanging though, rewriting the build-system for something as complex as the kernel would be quite the undertaking even just from a technical standpoint, let alone that you now have to change thousands of peoples workflow by having them install and use a new build-system (some of those people are probably dedicated to just maintaining the build-system too, so their job would radically change) as well as figure out a suitable build-system in the first place (which one would work well for the kernel? which one works on all the architectures people want to compile linux on? etc)

B from linus' perspective just means "wait a year for things to automatically get better (after throwing some money at it)" which seems like the low-effort solution.

Just needs someone to grab a Python Makefile parser (e.g. http://benjamin.smedbergs.us/pymake/), a python KConfig lib (e.g. https://github.com/ulfalizer/Kconfiglib though I guess this isn't really necessary) and write a ninja generator for the kernel build system.

Web dev here. There should be a gulp plugin for that.

shudder. Node.js-powered linux kernel builds? One small step towards https://www.destroyallsoftware.com/talks/the-birth-and-death... coming true ;)

"Just". The "mostly" in pymake's tagline (below) is the tricky bit.

pymake: A mostly GNU-compatible python implementation of `make`.

It's simply that doing that many system calls takes that amount of time. Even though everything is hot in the cache, make still needs to call stat on every single file in the kernel source tree, to detect those files that have changed.

The suggestion in the comments to allow a Windows-style batched stat is a good idea. This means that more work is done per system call, reducing the number of calls and therefore the amount of time spent saving the CPU state to switch to kernel mode, and then restoring it again to switch back.

The system calls are plenty fast. As I mentioned, we got Chrome's build of comparable size to around a second, and it still stats every file.

(A batched stat would be nice too, but it's not the most critical thing to optimize.)

Linus says it is about 5% of the time taken in his run. Well, 5% of 30 seconds is not too far off a second.

But yes, the kernel build is doing all sorts of stuff other than stat, like starting shells and sub instances of make.

Linus says about page fault overhead, not system call. I understand this is a much different thing. On x86 page fault raises an interrupt. For a system call there is an instruction designed for fast control transfer to avoid costly interrupt

> The system calls are plenty fast

No they are not. Cutting context switches in system calls is one of the easiest ways of boosting throughput in your average unoptimized Linux app, in my experience. Exactly because so many developers go around ignoring the cost of system calls.

Reply on news.ycombinator.com