> But you've mentioned a bunch of scripting languages, and CL was never competing with these.
Except that it WAS. Programmers have a limited amount of mindspace. So, the question as a programmer in the late 1980's/early 1990's is how I'm going to allocate that space:
1) Systems language--C is pretty much your only option at that point unless you are in assembly (God help you). Anything bashing hardware (graphics, audio, networking, etc.) goes straight for C.
2) Many programmers are also system administrators in this time frame. So, I'm going to use a language that mangles text well--Perl wins here. People transition to other languages only once they get fed up with Perl.
3) Operating system specific languages. Unix pushes you toward C. Windows was C or Visual Basic (that ecosystem was huge at one point). Mac was C or Pascal.
So, in this time frame: Lisp is painful for system programming on small systems (fails 1). Lisp manipulates strings badly (loses on 2). Lisp is not tied to an architecture that would drive adoption (loses on 3).
So, as a programmer, Lisp is, at best, at slot 3 or 4 in terms of programming languages for daily use. Is it any wonder Lisp never got real traction?
You also have to consider how knowledge got disseminated in this time frame. The web doesn't exist. Email is actually scarce unless you are at a university or a big technology company. Netnews is something that soaks up enormous amounts of disk space so sysadmins hate carrying it in useful ways.
It is hard to appreciate, in this era of instantaneous, ubiquitous information, just exactly how much the pink camel Programming Perl book with its cookbook section singlehandedly drove adoption of Perl. I had 3 copies of that book that I used until they disintegrated. Can you think of ANY other programming language book for which that is the case? (Actually, I can, but I'm the odd duck. My workbook copy of The Little Lisper got the same level of abuse to disintegration).
> The unstructured text-oriented philosophy is frankly insane
It is NOT insane when memory and disks are small--which defines most of the time when Lisp was popular. Structured text requires lots of extra overhead that simply wasn't going to fly when RAM, disk, and bandwidth were very restrictive.