What's wrong about the statement? The black box algorithm might have been generated by machine learning, but it's still a computer program in the end.
Like look at our brains. We know decently well how a single neuron works. We can simulate a single one with "just a computer program". But clearly with enough layers some form of complexity can emerge, and at some level that complexity becomes intelligence.
It isn’t a given that complexity begets intelligence.
The suspicion is that they are good at predicting next-token and not much else. This is still a research topic at this point, from my reading.
They're obviously intelligent in the way that we judge intelligence in humans: we pay attention to what they say. You ask them a question about an arbitrary subject, and they respond in the same way that an intelligent person would. If you don't consider that intelligence, then you have a fundamentally magical, unscientific view of what intelligence is.
Which one of these comparisons you want to use depends on context.
The same seems entirely possible for current LLMs. On the one hand they do something that visibly seems to to be the same as something humans do, but on the other it is possible that the way they do it entirely different. Just as with the bird/plane comparison, this has some implications when you start to dig deeper into capabilities (e.g. planes cannot fly anywhere near as slowly as birds, and birds cannot fly as fast as planes; birds have dramatically more maneuverability than planes, etc. etc).
So are LLMs intelligent in the same way humans are? Depends on your purpose in asking that question. Planes fly, but they are not birds.
The same goes for LLM and human thought.
Flight (like "intelligence") means more than one thing. Planes fly, birds fly, but they not only use a different mechanism, they can't even do the same kind of flying that the other does.
Sometimes, the difference doesn't matter. Sometimes it does. Same for "intelligence".
LLMs obviously display what everyone prior to 2022 would have called "intelligence," before the goalposts started rapidly shifting with the release of ChatGPT. They can carry conversations about arbitrary subjects, understanding what you're asking and formulating thoughtful answers at the level of a very smart and extremely well educated human. They're not identical to humans (e.g., they don't have fixed personalities), but they display what everyone commonly believes to be intelligence.
Whether or not LLMs are intelligent (I think they are more intelligent than a cat, for instance, but less intelligent than a human) isn't my argument.
My argument is that complexity in and of itself doesn't yield intelligence. There's no proof of that. There are many things that are very very complex, but we would not put it on an intelligence scale.