1. The first version of TeX was written in 1978 and spread like wildfire in a couple of years: ported by others to 200 programming environments i.e. (OS, language) pairs! So when he started writing his new version in 1980 trying to be portable to all those OSes, he was basically forced to use Pascal, which was at the time in widest use at universities for teaching, and thus most widely available on computer installations. (C was still confined to Bell Labs and maybe a few other places.)
2. Tony Hoare had suggested that the source code should be published as a book. This raises questions of how best to present messy real-world code written with real-world constraints (rather than "clean" toy examples).
3. Pascal, especially the subset of Pascal he was targeting (the common denominator across Pascal runtimes/compilers in use at the time) was extremely limited and had poor support for strings, array parameters, even control-flow (no "break" and "continue" in loops, and goto "labels" had to be numeric); moreover doing anything nontrivial practically necessitated many global variables that had to be initialized at the start of the program. I've written a bit more about this at https://shreevatsa.github.io/tex/program/pooltype
4. This is a joke, but he was/is also annoyed by the anti-goto moralists championing their vision "structured programming" (pejoratively implying anything else is unstructured), so he called his vision "literate programming" (as no one wants to be accused of writing illiterate programs).
So after some experiments he was led to the idea of a macro preprocessor, with sections that would be reordered in the way needed by the compiler, but could be written in the order the programmer wanted to write or present them.
As for monoliths, partly that's how Knuth seems to think. And he seems to have no trouble keeping a lot of program detail in his head that would overwhelm most of us, and he has always been a machine-level programmer at heart and is acutely conscious of wasteful stuff like—a real concern at the time—subroutine overheads. Even in this interview he mentions things like APL versus his preference of languages closer to what a computer actually does. And partly the Pascal subset he was targeting dictated the program structure: the compiler should get all its code in a single file the way toy programs are written, like
var <declare lots of globals here>;
<declare lots of functions/procedures here>;
begin
<initialization code>
<rest of "main">
end.
So that's a bit about the genesis of literate programming. Now about some of your other comments:
- Your quote about top-down programming from literateprogramming.com is from the author of fweb, but actually Knuth himself doesn't advocate (or practise) top-down programming. One of the things he likes about literate programming is that the program can be written in a mixture of bottom-up and top-down programming without being forced to commit to either; from studying many of his programs it's clear that his preference is to build things mostly bottom-up, though each small "layer" he may write top-down and sometimes he may switch it up. (For example, in his TeX program he starts with "chapters" on string handling, arithmetic, memory allocation, data structures etc, building up to the main program as the climax of the book. Each "chapter", which is sometimes just a single function, is written top-down, more or less.)
- You linked to how the PBRT book introduces literate programming, and said it shows noweb/LP is "still too mired by constraints of C", but to be clear, it's just that the example they used for the idea of rearranging sections is one of overcoming some constraints of C using LP. (It's not really specific to C though; the same thing would apply whenever one had a lot of initialization to do in different places, like the giant object constructors one can see in many C++ codebases.)
- Tastes are subjective, but IMO the program he wrote for Bentley's question of "top k most common words" (not his choice of problem!) is actually very interesting: it's an exposition of an ingenious data structure called a (packed) hash trie that has, to the best of my knowledge, never been described anywhere else, and which he probably made up on the spot for this problem. (Packed tries are described in an exercise in TAOCP and the TeX program uses them to store hyphenation patterns—his student Liang's thesis was about this—but packing them randomly using an open-addressing hash table seems to be a twist particular to this program.) I was planning to write a blog post last year illustrating this with pictures and all that, but never got around to it. I did write a translation into (non-literate) C++ here: https://codegolf.stackexchange.com/a/197870
- Returning to a couple of your other comments and the phrase "top-down style associated with literate programming": I think another way of looking at this is historical. When Knuth learned to program, in machine code on an IBM 650 (I say this was the first and last generation of truly self-taught programmers) and later writing compilers in machine code (or with his own assemblers) for other machines, even the idea of subroutines (with standard calling conventions and all that) was hardly well-established, which is why TAOCP Chapter 1 has, under 1.4 "Some Fundamental Programming Techniques" an explanation of what subroutines are, followed by coroutines and interpretive routines. When "Structured Programming" took over in the 1970s (just the idea of using if/while/for and subroutines instead of arbitrary control-flow jumps), it was a revolution, and even that Knuth is not still fully on board with. (If anyone is interested I'll try to string together a blog post on all mentions I've seen of Knuth's opinion on "goto". Some juicy bits, e.g. https://twitter.com/svat/status/885344334735745024 https://twitter.com/svat/status/913114286951505920 https://twitter.com/svat/status/1336935057877991424.) Since then, the programming / software-engineering world at large has settled on certain ideas of structuring programs, in order to keep complexity manageable: the use of abstraction, encapsulation, modules, information-hiding, objects, avoiding global variables, etc. In some sense what Knuth came up with for himself with WEB/CWEB is a wholesale alternative to all these ways of structuring programs, bypassing the mainstream. This seems to suffice for him, but for the rest of us it's a highly unfamiliar/unpalatable way of writing programs. This doesn't mean "Knuth is doing it wrong" in the sense of Kartik's post, because what you should take away as "Literate Programming" from reading his programs is not the diff w.r.t. how a modern programmer would write it, but the diff w.r.t. how he would write it without LP. That is, if you take for granted that the program is going to have very few functions, lots of global variables, etc., the question is whether LP is a better way of writing or presenting that program than non-LP. (It seems the answer is yes, though even this is debatable. Besides, Knuth does say that without LP he'd probably write more levels of functions but with LP he doesn't "have to", so you may disagree with the question too.) You should instead look at examples of programs that a modern programmer familiar with modern idioms wrote with and without LP. I imagine these days one might have better results searching for people using org-babel (rather than cweb/noweb), say. I see also there's https://github.com/AndrewOwenMartin/nestor mentioned in another thread above, or "nbdev" that came up recently (https://www.fast.ai/2019/12/02/nbdev/ , https://github.blog/2020-11-20-nbdev-a-literate-programming-...).
- But that apart, returning once again to Knuth's own literate programs. I have struggled a lot with them. Many people have easily understood much more than me, but I suspect I have struggled more than anyone :D I had some grand plans to make it more readable etc (https://shreevatsa.net/tex/program), but eventually they started making sense. (I should update those pages…) The point is, literate programming still does not mean writing code in a vacuum for readers who know nothing. Knuth does not consider explaining basic language features (as you pointed out with the INFORM example in another comment here), and moreover he also assumes familiarity with his LP conventions (the overall structure of the program, how to use the index and flip back-and-forth) and even I think his usual programming conventions. So the fact that the ADVENTURE program has the source file's outline (with #includes and everything at the top) on the second page, with a typedef of the "boolean" type and a couple of general-purpose variables declared at the top of "main" (which will probably be (ab)used for lots of different purposes, I expect)—all of these are not a serious problem, for the "literate" reader. To Knuth, literate programming does not mean belabouring and explaining everything; it's just a way of structuring your program. And though he mentions reading the program like a book, he doesn't think of books as things to be read strictly word-for-word in the sequence they appear; I expect he imagines reading any book non-linearly (looking up the index, flipping back-and-forth, skimming, etc) and that's how the programs are to be read too. The order of sections in the literate program is only more or less the "right" one, not strictly.
About the ADVENTURE program in particular, which is a useful case study, let me post a separate comment (as reply to this one), as this is already very long.