>>I'm not convinced human intellect isn't doing a similar thing with math when we are doing it fast.
Yes, this may be somewhat similar to the LLM's method, depending on how close we want to consider long-ago forgotten acts of memorizing multiplication tables but where the lesson still 'stuck'. Maybe that's simply regurgitating pretrained closeness vectors, or maybe it's a table lookup.
There's three things we need to look at: 1) good results, 2) errors or hallucinations, 3) underlying technology or methods of calculating.
If we look only at results, we can easily get fooled. That said, I agree that even a non-intelligent box that still yields reliable results can be enormously useful, have massive impacts, and be massively dangerous in the wrong hands. Of course, we already have such tools in photography tools that are vastly faster & more accurate than any artist and computers that make make larger & better calculations, etc..
Looking at the errors and hallucinations, such as the arithmetic or single banana failure, we can easily see that these generative AIs fail any time the right answer would require wielding even a slightly abstract concept. They haven't got it.
This is not to say they couldn't get it with more layers of some sort to extract & wield the concepts, but that's a different machine. Two months or two decades or two centuries away? Who knows?
Plus, the internals of the generative AIs tell us the same thing - they are massive collections of vectors of likely associations of words or bits, and are returning the most likely set, wrangled a bit by grammatical or artistic filters.
So, while results look good, both the errors and methods indicate a lack or intelligence.
Also, this is the same trajectory of other "AI". It was long thought that making a world-class chess computer would achieve intelligence. Same for Go, or Jeopardy. All these were achieved, everyone was impressed, and then everyone noticed that it was just a big computation engine.
I do expect that we'll get there someday, and that Bostrom's Superintelligence explosion will be a serious concern. I even thought that ChatGPT might be approaching it... but then I started using it and reading more about the internals. We're safe on that lineage, but who knows what others are working on?