Seems completely nonsensical. Yes, neural networks themselves are not unit testable, modular, symbolic or verifiable. That’s why we have them produce code artifacts - which possess all those traits and can be reviewed by both humans and other machines. It’s completely analogous to human software engineers, who are unfortunately black boxes as well.

More broadly, I’ve learned to attach 0 credence to any conceptual argument that an approach will not lead somewhere interesting. The hit rate on these negative theories is atrocious, they are often motivated by impure reasons, and the downside is very asymmetric (who cares if you sidestep a boring path? yet how brutal is it to miss an easy and powerful solution?)