> Being able to query the latent space of a large subset of human knowledge using natural language is the closest to 'intelligent computer' as we've ever come to.
I would agree with this statement, but it's still laughably far from it. As the other comment said, it lacks the ability to rank the veracity of the information it presents: it will present it in a grammatically correct way, but it has no way of knowing if the information it's presenting is accurate which explains ChatGPT's tendency to just make some shit up when you ask it things.
And in the odd event it is correct, it's simply digesting information from other sources, likely search engines either it's own or not. Again, this is not useless and not not-clever: however in that way it does remain pretty useless because of the above issues of accuracy. The best it can do is get you the most highly ranked information on a topic based on it's pool of knowledge, and that may or may not be the most correct one.
Also I'd add: these two issues compound one another, because the LLM is obscuring the source of it's information. You can have it add the link to it's citations, that would be helpful, but you don't know where it got those citations or why it picked them, and the LLM can't explain that for you. And, those choices were based on factors that are definitively not the accuracy or reliability of those sources, merely their ranking in whatever search algorithm is being used. Even if you presume this AI was trained on data relevant to the topic you're asking about, that doesn't mean it (and in fact, as I understand LLM, it's basically incapable of) understands that topic in a way that makes sense to a subject matter expert.
If you asked an engineer what size lumber would be required to construct a floor that could support a small vehicle, that engineer would probably have a rough guess based on their previous experiences, and further, could then do the work: research the weight of the vehicle, the situation the structure will be installed in, the environment it will exist in, etc. and come back with a solid answer to your question. An LLM, by comparison, would consult the writings it was trained on for similar questions, and create an answer it thinks matches yours. But what does "similar" mean here? Is the wood and the dimensions the same? Is what it's looking at based on a small vehicle or a cement truck? Is it looking for articles about structures in similar climates or did it pick some that were closer matched to the structure but in completely different environments? On and on.
Using an LLM this way takes the problematic aspects of googling things and basically factors them out, because now you can't see what it's googling, which results it's picking or why, and can't verify their relevance. How many times have you searched specific things and left a google result completely befuddled as to why on earth it included it? Now imagine that, except there's essentially zero chance the AI caught that, and it incorporated that irrelevant data anyway.