We are already in a world where neural networks are used to drive safety critical processes, and engineers are having to reason rigorously about the overall behaviour of systems that include components behaving in ways that cannot be simply understood or enumerated - because if they could be modelled with simple logic, the engineers would just write and use that logic instead of training and incorporating a neural network into the design.

You deal with problems in this space by treating the neural network output as yet another noisy signal that is fused like any other to drive your comprehensible, rigorously designed system with its restricted range of behaviours that can be reasoned about and made to fail safe.

It feels like there is yet a great deal of room to extract utility from AI with this sort of approach - keeping it in a box which can only interact in narrow and well understood ways with the outside world - before one starts hitting the limits of its utility.