Walter here (author of Warp). AMA about Warp.
I'd like to learn more about this. I spend a fair amount of time building on HPC systems. Frustratingly, compiling on a $100M computer is typically 50x slower than compiling on a laptop due to the atrocious metadata performance of the shared file system. Configuration is even worse because there is typically little or no parallelism. Moving my source tree to a fast local disk barely helps so long as system and library headers continue to reside on the slow filesystem. A compiler system that transparently caches file accesses across an entire project build would save computational scientists an enormous amount of time building on these systems.
First off, a text preprocessor is a classic filter program, and ranges-and-algorithms is a classic filter program solution. Hence, if that didn't work out well for Warp's design, that would have been a massive failure.
The classic preprocessor design, however, is to split the source text up into preprocessing tokens, process the tokens, and then reconstitute output text from the token stream.
Warp doesn't work like that. It deals with everything as text, and desperately tries to avoid tokenizing anything. The ranges used are all ranges of text in various stages of being preprocessed. A major attempt is done to minimize any state kept around, and to avoid doing memory allocation as much as possible.
Warp doesn't use much of any classic algorithms. They're all custom ones built by carefully examining the Standard description of how it should work.
What are the main stages/algorithms that constitute the processing pipeline?
I'm mainly trying to understand the high-level roadmap/design enough that I can use the source code itself to answer my more detailed questions. :)
http://gcc.gnu.org/onlinedocs/cpp/Traditional-Mode.html
The other major area of difference seems to be that pre-ANSI people used foo//bar to paste tokens (whereas now we use ##). If we're talking about C, that's easy to update; apparently some Haskell folks can't do that for their own reasons. Again it's a use case which is not preprocessing of C or C++, so it seems OK to ignore it if you're implementing a C preprocessor (as opposed to a generic macro expander usable with Makefiles and Haskell).
Warp does support the obsolete gcc-style varargs, but other obsolete practices it discards.
In any case, I haven't seen any Makefiles big enough to benefit from faster preprocessing.
[My first real project used Zortech C++ for OS/2, thanks for the fond memories...]
I'm credited with a few innovations (like NRVO), but for the most part I just put together novel combinations of other peoples' ideas :-)
On a related topic could you tell us about coroutines/fibres in D ... How are they implemented ? Can they be called from C ? are they are used in D's standard library, examples of a few notable use cases (I guess one would be Vibe.d), examples of asynchronous idioms in D.
Since that's a lot of questions, pointing to documents would be fine too.
..and thanks for answering questions here.
The D runtime library does have fibers:
Other than the speed processing, is there anything about how Warp operates that would make it easier to implement a distributed build system? It seems like the preprocessor can sometimes play a role in whether builds are deterministic enough to be successfully cacheable (__FILE__ macros and such).
The same should work for any other compiler, provided they don't use preprocessors with custom behavior.
I don't see any reason why Warp cannot be used in a distributed build system.