The appealing thing about this is the programmer doesn't have to understand anything. If you have little data, approximation just isn't going to capture the situation. Either the programmer gets an understanding of the system (extremely costs and time-consuming) or we create systems that are themselves capable of this understanding. But no one knows how to do this, all the "artificial intelligence" victories anyone has observed have come from throwing computing power at a problem. Maybe someone will figure out how to throw computing power at the general problem of understanding but I'm doubtful.
I mean, there are a few situations that can stay pretty constant but just think when a bunch of AI system are established, all thinking the world won't change, then a cascade of causes changes and the systems no longer working becomes part of that. Of course, we already saw fragility of modern systems with Covid, so let's add even more!