It seems like our points of view are closer together than I had thought.
I first learned about Lisp because my uncle lent me a Lisp textbook; later, an executive at the company I was working at lent me SICP and exploded my brain. Most actual Lisp users I've met in real life weren't reading comp.lang.lisp either. But it's true that most of my contact with Lisp has been over the internet.
That's because most of my contact with any technical discipline has been over the internet, because almost everyone's has been, for 30 years now. It would be inconceivable to say, 25 years ago, "Most actual Perl users I've met don't read comp.lang.perl.misc," or, "Most actual Python users I've met don't read comp.lang.python," but it was and is true of Lisp. Sharing code and discussions over the internet has been a superpower for the programming-language communities that have managed it, including cross-language communities; I first got to know pg on the Lightweight Languages mailing list, for example. Meanwhile, Lisp users were isolated in the kinds of tiny single-implementation ghettoes you describe, or were collaborating only with other researchers at the same institution.
What I'm pointing to is precisely the fact that comp.lang.lisp was just mostly random people with little impact on what actually happened in Lisp. Every person who might have taken up Lisp, who instead focused on Perl or PHP or Haskell, was a missed opportunity for Lisp to thrive. As with the undiscovered antibiotics resulting from our current pharmaceuticals regulatory regime, the costs are invisible, but staggering. They are the Lispers that weren't born during the 01990s, the Lisps that didn't get finished or even started, the libraries that got written in Perl or JavaScript because Quicklisp and Leiningen didn't exist yet (as far as I can tell, there's still no equivalent for Scheme.)
You're right that trolls like Xah were what made comp.lang.lisp unusable. However, despite the almost 800 lines of code he contributed to Emacs, Naggum was one of them—the worst of all. Those 800 lines are apparently the totality of his public code, and in total they're smaller than some of his individual hate-filled posts to comp.lang.lisp. The period of Naggum's unquestioned domination of the newsgroup was precisely the period during which Usenet was the most important groupware medium. And the practice of establishing dominance by public ridicule didn't end at the boundaries of Usenet; as dang points out above, to this day we still have to combat it in virtually every public discussion of Lisp. That's Naggum's lasting legacy.
Since he became inactive, though, the situation has improved dramatically. We have Arc, Clojure (and Leiningen!), SBCL (which started while Naggum was still active but took years to become popular), the entire Racket empire (now built on the redoubtable Chez compiler), several new "Little" books, miniKANREN, Hy, Quicklisp, R7RS-small, and, as you point out, reasonably functional subreddits. #Lisp on Libera is a pleasant place, and Emacs Lisp and Guile now have native-code compilers. ACL2 and PVS are significant minorities in formal methods. You can even run PDP-1 LISP, or modify and reassemble it with MIDAS.