> cascade of architecture-specific functions is done as ifdefs within one file, so that it's easy and obvious and clear what's happening...

Not in my opinion, having an arch ifdefs makes things much less clear as I effectively have to parse the ifdefs within the code while trying to understand what it's doing, compound that with n arch specific optimizations and it quickly grows out of hand. An arch over-ride file (e.g. $arch/foo.[cs] over-rides dir/foo.c) is much easier to read.