I think you misread the philosophy he talked about. He actually thinks that having the problems about threading, libraries and low APIs is a good thing, as it helps us maximizing social value of software we develop.
Python in Vim has slowly become the go-to choice to talk to an external program from Vim, because VimScript is simply that bad.
I know little, however, about how hard ELisp makes it to interface with an external program. The examples he gives (a JS interpreter in ELisp, and Semantic) lead me to believe that the authors wrote all that in ELisp because it was easier than to play with an external program. Am I wrong?
Will Guile ease the task of interfacing with an external program?
I suspect people who write packages entirely in elisp simply like elisp. And why not? The language itself is mostly OK to use, even if it doesn't always conform to modern standards.
Writing elisp is not hard. What's hard is integrating all the modes, keymaps and quirky behavior of coupling all these smalls bits of code together. This is what makes emacs hard. Emacs, the "core editor" itself, is incredibly small. Way smaller than vim (build it by hand if you don't believe it). Emacs as the editor that you use every day is almost pure extensions.
There's a /whole lot/ of global state in an editor. This is not going to change with Python, Guile or any other language. Users that complain about ELisp, most of the time actually never tried to write ELisp at all (they just glue some elisp around in .emacsrc). That's fine, but they don't realize what's going on under the hood. The C/C++ mode in emacs is completely different to the "stupid" syntax highlighting in Vim or most other editors.
I normally recommend switching to another editor if after several years you still don't appreciate the difference. IMHO emacs, as it is, is a great editor. ELisp is part of the success (though guile would be a step forward).