[NOT THE OP]
>> All you see is numbers and functions, and all I see is problems being solved
by these AI/ML methods and often in quantifiably better ways than intelligent
humans can.
Maybe in playing chess and go, but in any other task that neural networks have
been compared to humans directly, the ability to outperform humans is more
informative of the limitations of the measures of performance, than of the
capabilities of the tested systems (typically, deep neural networks).
Take large language models (LLMs) for example. Last time I looked -it was a
while ago but- LLMs like BERT, ERNIE and their famous friends scored very highly
and even above humans in natural language understanding tasks, such as those in
the GLUE and SUPERGLUE benchmarks.
You'd think that a system that scores 85% in a language understanding task has
developed at least _some_ ability to understand language. And yet this is not
the case. LLMs, despite their ability to score highly in formal benchmarks,
remain dumb as bricks and as incapable of understanding human language as a
rock.
We know this because every once in a while someone in the field of NLP looks
away from their favourite leaderboard for long enough to ask serious questions
about the performance of systems, as for example [1] who started by asking why a
language model like BERT would be any good at argument comprehension (BERT
scored only three points below humans in a relevant benchmark task). Everytime
anyone undertakes this kind of analysis of the abiltiy of LLMs to understand
language, the conclusion is that they don't, and instead they learn "cheap
tricks" and overfit to statistical regularities of the benchmark datasets.
Similar analyses have found similar results in machine vision (e.g. [2], [3])
etc.
The question then is, in what sense, exactly, is an automated system solving
problems "in quantifiably better ways than intelligent humans can", when that
system is not even solving the problem it's supposed to be solving, like the
problem of understanding language by LLMs? The idea that those systems "solve
problems", when they 're really only overfitting to their training sets, is an
artifact of the shoddy scholarship that is typical in deep learning research,
and the OP is right to be concerned that there is little understanding between
machine learning researchers of what, exactly, it is that their systems are
doing- or even trying to do.
And, as the OP points out, I'm afraid that really is the result of a poor
background, not only in sciences and mathematics, but in AI and even machine
learning itself, among the current generation of machine learning researchers.
Contrary to what the OP says, though, this paucity of relevant background is the
result of an influx to the field of people from the sciences, who have a
background in continuous mathematics, but not in AI or computer science. It
is, indeed, a background in computer science that can best guard against
misconceptions about computers being wonder machines with magickal intelligence
abilities. At the very least, you don't have to explain to a computer scientist
why solving one instance of a class of problems does not mean a general solution
for the entire class. That comes built-in to most who have graduated in the
discipline. But is it built-in to phycisists, or biologists, who study very
different systems?
________________
[1] Probing Neural Network Comprehension of Natural Language Arguments
https://aclanthology.org/P19-1459/
[2] Intriguing properties of neural networks
https://research.google/pubs/pub42503/
[3] The Elephant in the Room
https://arxiv.org/abs/1808.03305