What a weird take. These AIs pretty obviously so work. Sure some parts are surprising, more to me than to experts of course, yet why not take the less understood parts and try to understand them, experiment, see what works and why and how, see what doesn't, that's science, not "boy geniuses".
its only a weird take because the scientists who "do actually know how it works", are lopping this in as a software product. And the softwe=are engineers are still thinking they will be able to take credit for it. (wont be able to fix it)
Applied math and software have nothing to do with each other. Any more than applied chemistry (reduced to math) written in a Visual Basic Excel macro is software.
It aint computer science just because you typed it in.