It's a hard problem, as evidenced by the inherent complexity in autotools, but there are a lot of smart folks who could apply their effort towards that goal.
The problems broadly speaking are:
* Assumptions: autotools assumes that only a basic Bourne shell is available, and a basic make, and that make is a good way to build projects. All of those are questionable, and certainly if you get rid of the assumption that you're only allowed to use ancient tools, you could immediately make autotools better, if only by removing all the shell cruft. But then someone will complain because your project doesn't compile on SunOS 4 or whatever.
* Familiarity: When I'm packaging your project for Fedora, I will not be pleased if you've used some obscure or half-assed build system. If you're using autotools, I may dislike it, but at least I can fix things.
The second one is a difficult one, because until your build system gets popular, it will never be familiar, and us packagers will keep cursing you. Once it's popular, it'll probably be quite crufty because it has had to handle all the horrible corner cases that autotools can do now.
Look at the nginx configure scripts. They're hand-written, and modularized. Check for "foo"? There's a shell script. Autotools does the same thing via special-purpose autogenerated shell code.
That's why the configure script is 1000's to 10's of 1000's of lines of shell script. It's all auto-generated crap.
And autotools doesn't have the concept of bootstrapping. Building a C program? Great! Test everything with shell scripts. It might be more efficient to have C programs doing work, but no... it has to be the lowest common denominator.
Because managing multiple pieces is too hard. Why? Because there's no modularization.
I've used autotools for decades because everything else is worse. But I have a long-term deep-seated loathing of it which is hard to put into words.
I take it you've used CMake - could you explain in which ways you think it's worse than autotools? I agree it has concerning design flaws, but they don't come anywhere close to autotools' and I'd say CMake is overall dozens of times saner.
The difference is that autotools generates source distributions that do not need autotools to build, while a cmake project always requires cmake when building.
Charmonizer needs a supported C compiler command line interface (POSIX cc, MSVC, maybe others in the future) which can run "system()" commands in a supported shell (sh, cmd.exe).
The testing logic is all implemented in C code. Instead of shipping a shell script, there's one big amalgamated C file, plus a tiny "configure" wrapper to present a familiar interface.
Charmonizer hast given Lucy Autoconf-style probing on all our target platforms without adding any prerequisites beyond what Lucy already needs itself. This has made installing Lucy much easier for a wide range of users.
Right now Charmonizer remains a private component supporting only Lucy but if there were to be interest it could be broken out as an independent project at Apache.
I realized this a while back, when I was advocating git at my company (maybe 5 or so years back...) some developers loved because they like the idea of it, they wanted to know how it works underneath etc., other group didn't care, didn't want to hear about internal they just wanted a list of commands to type to get their code committed.
Neither one is wrong or or bad, it just comes down to personalities or interests. Autotools is a bit in the same area.
A lot of people will complain about it, because they want to use it to get stuff done. But they will never the ones to start writing tools like it, because they do something else -- databases, graphics, numerical simulations etc.
So "start working on a successor if you don't like it" might not have the same effect for this project.
For example, the SCons wiki has some interesting comparisons.
https://bitbucket.org/scons/scons/wiki/SconsVsOtherBuildTool...
Ubiquity is important for a build tool.
E.g.:
The script detects just what is necessary for the program and is easy for me to understand, extend, port and debug.
You can fix it right on a given target platform where you're trying to get it working.
You can't do that with an Autoconf configure script (in a way that you can merge back to your stream), because it's generated. You need an installation of Autoconf.
If you want to patch the Autoconf-generated configure script with minimal changes, just reflecting the topic of the change you're making, you need the exact same version of Autoconf with which it was generated, otherwise there will be irrelevant diffs all over the script.
(I learned this when I developed an embedded, cross-compiled LInux distro from scratch. On my development system, I had about half a dozen installations of different versions of Autoconf, so that I could create minimal patches against configure scripts!!!)
And, of course, you first have to work backwards to figure out how to coax the script behavior you want indirectly through the m4 macro files. Forget about "I just want to make this tiny tweak in one command and it will Just Work".
In my script, you just ... open the script and write the necessary code. Then regression-test it on the other targets, do any necessary tweaking and commit.
Once you have written one of these scripts, you can easily re-use it. My above script is BSD-licensed. Anyone can take it and cut out whatever they don't need. If you want help using it or have any questions, you can fire me an e-mail.
End-users don't build from source in the first place. At least, if they do, then the developers and/or packagers have failed to do their job.