he lost me at "a language model’s vocabulary is limited to the words that exist within the model’s training texts, which means a LLM can only refer to objects and relations that we humans have already discerned, named, and written about."

This is trivially demonstrated to be a false statement, as GPT is capable of synthesizing entirely novel words based on very little input guidance.