And the problem with current approaches is entirely that while we can "train" a NN to produce specific emergent behavior associated with the training data to a great degree of accuracy, we're totally unable to demonstrate that these systems have perfectly(100% accurate) predictable behaviors.

This is because we're totally unable to come up with a coherent model for why the emergent behavior occurs given the input data and NN training. We know how individual elements of the NN work, and we can describe how the training system works. But the whole notion of repeatedly letting perturbations in a control value or control values dictate the entirety of the performance of a system is nuts.

The only way to determine how a NN will perform in a given situation outside of the training set is to actually feed it the stimulus and check the outputs. Given the amount of stimulus that we as drivers routinely get, it's impossible to say with any degree of certainty that a self-driving car that is built on an NN classifying engine will accurately classify everything in all situations and lighting conditions, because it's impossible to feed it a training set large enough to encompass those situations.

That leaves the question of making a classifier that is better than humans. And whether an NN is better than a human depends very much on the situation. We could make some statistical arguments, but when you're gambling with peoples' lives here it becomes difficult to tolerate such arguments. It's easy to be blase about it until it's your child chasing a ball in front of an AV, at which point any discussion of the statistics is academic.