Chromium Notes: Ninja, a new build system
neugierig.org
neugierig.org
https://github.com/apenwarr/redo#readme
(previously: http://news.ycombinator.com/item?id=2104803)
I've personally managed to get the redo test suite to pass on Win32, running inside a port of BusyBox: https://github.com/pclouds/busybox-w32
Yes, it's an extra download, but a 500kb standalone executable is much nicer than the hundreds of megabytes required by Cygwin or MSYS.
I weep for the lost souls of those poor benighted developers still stuck on Windows. No, no wait. why should we care about Windows developers? If they're still stuck there most likely they are deeply committed to the Microsoft Visual Studio toolset, in which case they don't (or can't) care about what goes on in the rest of the universe, and they really don't matter. (By which of course I mean that as fellow human beings they matter, and of course I have sympathy for their suffering, but by their own choice they are locked in such an impregnable ivory tower that it is neither practical nor economic to try to break in and rescue them)
Other than that, who else uses Windows? Oh yes, you have corporate teams doing 'enterprise' Java. They'll typically be using some colossally retarded Ant build system that takes 7-30 minutes to run (sadly, I'm not even exaggerating for dramatic effect). The problem with Ant is it is so easy to just bolt on 'one more thing' that it rapidly evolves to some horrible beast of a thing that lumbers around sucking everything into it, Katamari Damacy style. Really, there's no help for them either. You can't for example suggest that blowing away the entire database and recreating it from scratch in order to run the entire set of automated unit tests is something best left to the continuous integration server, or that should be done once a day - maybe out of hours or at lunchtime. No no, it has to be done every time you do a build. And that doesn't even touch on the ginormous mess that is the enterprise components, where each application server has its own arcane and unholy rituals to create its beans from the blood of unicorns and tears of virgin developers... try messing with that abomination and you're in for a world of pain. There's just no helping them either, though in their case usually they want to be helped, but they are captive to the primary problem of enterprise development, which is that the people who choose technologies and mandate tools and processes are usually so far removed from the actual use of those tools as to be completely immune to the pain, and unable or unwilling to hear the wailing and gnashing of teeth of the programmers.
Don't even get me started on checkstyle with rules like no line can be longer than 80 characters (despite there being absolutely no good reason for this other than the horrible horrible UI of eclipse - in the absence of that you can easily get way more than 80 characters on the screen).
twitch
In a corporate environment, it is not unreasonable to assume that the developer can have a second monitor. Sometimes you have to gasp be nice to someone in order to get it, but that is not too onerous. Hence I believe your view is outdated; in practice horizontal is much cheaper than vertical.
When printing, one should probably do a few things in order to improve the appearance on paper anyway. Examples - you might want to set the indentation depth a little less than normal, say 2 spaces. You should also twiddle the font until it looks good, or is good for purpose (there are different reasons to print out code) and you might want to concatenate several smaller classes onto a single page, or remove the 20 line header of corporate pseudo-legalese from the top of each class, or even remove the imports, or not print the getters and setters. In other words, unless your purpose is to murder trees, you will likely hand-tune the printing to optimise it, at which time you might choose to set an 80 column width - if you felt that having each page consist mostly of a thin column of text pressed up against the right hand side of the page was the most aesthetically pleasing thing.
----
Run over lines do look a little bit ugly when printed I admit, this is true. However, since code is viewed 10-100x more often on screen than on paper, I believe it is inefficient to prioritise for printing (in a cart before the horse sense). Moreover, since the choice is between a little bit of ugliness when printed (inifinitely many chars per line) compared to a lot of ugliness when viewed on screen (80 char limit causing frequent line breaks, which are themselves heavily indented, which makes the run on itself more likely to spawn a run on), I prefer the lesser of two evils.
Anyone writing cross-platform software of any kind. In the context of this article, say, the Chrome team.
The same article which specifically mentions that most of them are running Linux? Or a different article?
And I wonder whether we can make a version of git that uses inotify.
If he's willing to endure a long-lived server process, he can probably have no-op builds with a tup-like system in less than a few milliseconds. (Basically as long as it takes to run through a single `if' and return to the shell; since no news is good news.)
It turns out (not terribly surprisingly, given how it works) that tup's overhead is very small.
> I had originally intended to make Ninja be memory-resident and to use inotify to keep the build state hot at all times. But upon writing the code I found it was fast enough to run from scratch each time. Perhaps a slower computer would still benefit from inotify; the data structures are set up such that using inotify shouldn’t be hard.
0.1 sec to recompile 100K files project. It looks really impressive.
It would be interesting to see what would happen if they were using waf instead of scons. Waf is also in python, and started as a fork of scons (but is so different that it can now be considered as a totally different design and codebase). Waf is much faster than scons (easily one order of magnitude), to the point that I think it would be hard to be much faster without losing features and/or system specific features (notifying systems, using checksumed file systems, etc...).
Samba has been using waf for > 6 months now, and they seem quite happy with it. As a former user/contributor of scons, I much prefer waf now, and anyone interested in complete build systems should look at it IMO.
— Ryan Dahl, in reference to Node.js
http://bostinnovation.com/2011/01/31/node-js-interview-4-que...
I have experience with quite a few build tools, from autoconf/make to waf, with custom ones, and waf is by far the one with the least WTF so far if you want to do something which is hard. It gives you the power of a real language, which is needed for complex builds IMHO. It looks like node.js is now using cmake, and its macro language is quite weak and error prone IMO, although it definitely works for non trivial projects. Waf is also fast, small enough that you can hack it if wanted (compared to cmake with C++ + architecture based on autogenerated makefiles...), and just enough usage by non-trivial projects I would expect for a tool I may depend on (samba and ardour are two quite big, multi-year, > 100 kloc of cross-platform, multi-language tools).
Regarded the specifically cited point of including dependencies on compilation flags, unless I am confused, I believe it can be done much more quickly in standard make, in one of two ways:
First way: make the build path of the object file dependent on the build flags. This has zero performance penalty, and also has the nice side-effect that when changing flags (e.g., from release to debug build and back again), you don't have to recompile everything, because you still have the previous build sitting around.
Second way: store the build flags in a separate makefile snippet (which you can either include or get the value of using $(shell)), and add that as a dependency of the object files. This has minimal performance impact since it's just another normal dependency for the object files. (This second trick is from one of the articles linked to about redo posted a few days ago; sadly I don't recall exactly which.)
I'm always interested in alternatives to Make because I just find it so painful. However, I'd say that only about half of Make-related pain comes from its dependency management. The other half, to me, is in using its language, and Ninja doesn't seem to do anything to ease that pain. Its manual says: "You should generate your ninja files using another program." That seems like a bad sign to me.
Tools like CMake can be helpful when there are lots of configurations available and dependencies to check, but on a small project I want to write a quick script that will just work. CMake and its ilk add another layer of complication that I don't want to have to deal with most of the time.
Does it mean that there's something wrong with the current state of affairs that you have to rebuild your infrastructure for a large project? Or does it mean that Google is so unbelievably great that everything is not good enough for them so if it's important they have to redo it from scratch?
That's just life with a large project.
If you have a build system which your users are also concerned about, readability and maintainability are a lot more important. SCons managed to achieve most of this by using a Python syntax. But its behavior can be quite unpredictable at times.
I was trying to make the point that among the operating systems doing builds under his new system, Linux was compiling the fastest. I was suggesting an interesting possible explanation for that.
Is it because its Not-Invented-Here culture?