Hacker News Reader

Comment by kazinator | Hacker News Reader

I ported undump from Emacs to GNU Make once!

I was working in an organization that developed a big network switch, with a large C++ application running on it whose non-recursive Makefile took 30 seconds just to load and parse all of the include makefiles throughout the tree, before actually building anything.

Half a minute of waiting just waiting for that second that it then takes to recompile a single .cpp to a single .o and link everything.

I got tired and added a "make --dump" option which used the GNU Emacs undump code to dump an image of make with all the rules loaded from the Makefile. Then "make --restart" would instantly fire off the incremental rebuilds. (Of course, any changes to the makefiles or generated dependency makefiles required a new --dump to be taken to have an accurate rule tree.)

Another idea would be just to add a darn REPL to make, so you can keep it running and just re-evaluate the rule tree.

View on HN

Isn't the ninja build system just a faster replacement for make. I use CMake primarily it lets me create makefiles, ninja files or vs projects. I noticed that in the general case ninja is 5% to 10% faster than make on builds that are more than few seconds long.

Then I found that if I did serious file manipulation at build time, like copy trees of files dependent on other thing in the build, I could have tens of thousands of targets, one per file usually. Ninja might hiccup for a fraction of a second on these shenanigans but make often sits and spins for 20 or minutes.

Unless you want to write the makefile yourself why not use ninja?

Because then I have to ask users to install ninja before they can build my program. In the projects I'm working on, I don't have any of the issues that ninja solves.

You can generate both. That gives you ninja for development (which had a problem you described) while the official builds still can use the makefiles that should result in the same output.

> You can generate both.

That means that a user who wants to patch the build rules has to have the generator, and learn the generating language instead of the Make language he or she already knows.

Autoconf has this disease. You can build from an official tarball, but touch anything (or use a git checkout) and you need auto-this to generate auto-that. Not just any auto-this, but a specific version, that is seven releases behind current, or else three releases ahead of what your distro provides.

> should result in the same output

It should; but someone has to ensure that it does. That's just another unnecessary concern that doesn't actually have to do with anything with the functionality of whatever is being built. We would like to spend our QA cycles validating the program, not three ways of building it.

Best to have just one way to build, and don't require users to install extraneous tools.

You've confused autoconf with automake. The output from autoconf is just a list of variables that are sourced by your handwritten Makefile, which you can supply yourself if you don't feel like executing autoconf.

It's automake that writes your Makefile for you, but you can just skip using that. E.g. the Git project uses autoconf optionally but not automake.

Rather, you might be confusing the ./configure script with autoconf (the tool which generates the script from "configure.ac").

I know the difference between autoconf and its generated ./configure target.

Your comment overall indicated to me that you were talking about automake, not autoconf. But if not, fair enough.

E.g. you talk about "learn the generating language instead of the Make language". I know you were using that as an example, but there's no general non-horrible replacement for autoconf that you can write by hand, as opposed to automake where you can write a portable Makefile

You can of course write a bunch of ad-hoc shellscripts & C test programs to probe your system, but this is going to be a lot nastier and buggier than just using autoconf to achieve the same goal.

You also don't generally need autoconf to build projects you clone from source control in the same way that you need automake (because that actually makes the Makefile).

The output of autoconf is generally just a file full of variables the Makefile includes, if you don't have that file you can just manually specify anything that differs from the Makefile defaults, e.g. NO_IPV6=YesPlease or whatever.

The Git project, whose autoconf recipe I've contributed to, is a good example of this. You can "git clone" it and just issue "make" and it works, but if the default config doesn't work then "make configure && make" generally solves it, but you can also just e.g. do "make NO_IPV6=YesPlease" if it was lack of IPv6 that was causing the compilation failure. It'll then get your NO_IPV6 variable from the command-line instead of from the ./configure generated config.mak.autogen.

The output of autoconf is generally just a file full of variables the Makefile includes

You may just be using a terminology I don't recognize, but like 'kazinator', I think you are missing a step.

The Git project, whose autoconf recipe I've contributed to, is a good example of this.

Great, let's use that as a specific example: https://github.com/git/git/blob/master/configure.ac. Line 2 says "Process this file with autoconf to produce a configure script."

I interpret this as saying that autoconf takes configure.ac as input, and produces a runnable 'configure' script as output. But you are saying that "the output of autoconf is generally just file full of variables the Makefile includes". How can these both be true?

I was using "autoconf" to mean both the software itself and all its output, including the generated configure script.

Confusingly, sorry about that, but for the purposes of discussing what software you need to generate the configure valuables you ultimately need when cloning from source control, it makes no difference.

Chains of discussion like this are why I like CMake. Much less confusion.

Few confuse the output of CMake with things that ought to be committed.

I like something called GNU Make for the same reason.

Few confuse the output of Make (namely your built program) with something that ought to be committed (like its source code or the Makefile).

> That means that a user who wants to patch the build rules

I think we have very different definitions of a "user".

I kind of see your point about parallel builds, but think a lot of it could be automatically verified with right CI system. Once it works, it works.

> Once it works it works ...

So long as nobody touches it, or any aspect of the environment it implicitly depends on.

Shake is usually even faster.

At first I thought I would have a hard time searching for it with such a simple name, but it clearly is a Haskell tool.

This looks like a reasonable alternative for people who control the whole lifetime of their code, life server software developers. But for someone shipping a library or anything they expect someone else to build, what advantages does this have over Make in the *nix world or CMake in general?

I wonder if most of those include files were the files auto-generated by the C pre-processor to track header file dependencies. In a toy benchmark[1] GNU Make spent 98% of its time processing those include files (for a no-op build). Ninja has a special optimisation for these files, where it reads them the first time they're created, inserts the dependency information into a binary database, then deletes them so it doesn't have to parse them in future invocations. AFAICT this accounts for most of Ninja's speed improvement over Make.

[1]: http://david.rothlis.net/ninja-benchmark/

Is this open source somewhere or company internal only? I think I could make good use of that also.

I updated the unexec code a bit in my project, but adding it to more projects sound like fun.

Well, it's GPL licensed which means they'd have to provide source code to anyone they give the software to. The only question is whether they're aloud to distribute it to anyone outside of their company.

Not contradicting you, but just posting to invite someone to correct me if I'm wrong. If I understand correctly, the GPL specifically grants a license to distribute to anyone you please. However, in the case where the software was originally distributed to a company, an individual working at the company does not have a license (because it wasn't distributed to them specifically) and can not redistribute the software. Only the company can redistribute the software (though, having done so, they can't stop a recipient from redistributing the software).

It's one of the really subtle points of the GPL and easy to get wrong (which I'm inviting people to correct my interpretation ;-) ). I often wonder whether the AGPL would work the same way because the "distribute" clause in it is awfully vague, from my interpretation. Just giving me access to the software over a network requires giving a license, it seems. So if they give me access as an employee, I should also get a license... maybe...

Please contribute that patch to GNU Make!

Reply on news.ycombinator.com