I'll never understand it when people quote a primary source and then summarize it in a way that completely ignores the original quote.

Olah's quote (edit - originally misidentified this as Pope Leo's) makes a lot of sense to me. He is saying, accurately, that modern AI (i.e. primarily LLMs) is created as essentially a mashup of our own language, and they are still a bit of a black box (or at least a gray box) to even their creators.

I don't know how you get from that to "he's framing AI as some new and fascinating form of consciousness".