Lack of knowledge could be another way of saying "not aligned" which is the core issue.
like, as a trivial counterexample, I can speak at 500 wpm. let's say that ChatGPT can generate 500 words per second for a single thread of conversation - I think that's a generous overestimate. now that's a 60x speedup over me, not a 1,000,000,000x like you're talking about. do you honestly believe they they can make ChatGPT run 16.6 million times faster, without changing hardware? do you think ChatGPT will just like, hit an inflection point where it realizes how to refactor its inference code to run 16,666,666x faster?
no, I think that's absurd. you're treating these things like black boxes but they are constrained by computational complexity, die area and the speed of light for Christ's sake.
AIs can scale horizontally as well as vertically.
Silicon transistors are much more efficient at computation than the human brain just as a jet engine is superior to to a peregrine falcon.
ChatGPT could absolutely generate thousands of words per second on existing hardware.