Hacker News Reader

Comment by DoingIsLearning | Hacker News Reader

> Traceability

A question for the AI Researchers and Engineers On the topic of _Traceability_ and as someone outside the field this is something that has puzzled me:

(Broad assumptions) Assuming a 'neural network' is a black box probabilistic model. How do we _prove_ in the context of _safety-critical_ systems that they will behave in a known determined and traceable way for all known cases?

I would be curious to understand the process/methods that people use in ADAS system or Medical Imaging (and now Defense applications as well) in order to provide 'verification' evidence for such a system? (Focusing more on verification/provability from a regulatory standpoint)

View on HN

There is work in progress for understanding the content of a neural network and avoid this "black box" effect. This is still far from reached but there are some advances. See [0] for example.

Also, remember that AI includes more than NN. You can use some other models, as a Linear Regression or a Random Forest which are perfectly explainable.

[0] https://christophm.github.io/interpretable-ml-book/neural-ne...

Indeed ML is vast and not limited to NN.

My question was more towards people who are actively using NN as a solution and _have_already_ shipped products in a 'regulated' industry.

The products are obviously out there, so to me it is confusing what evidence can a team provide at this point in time (with the current NN implementations) in order to get a product using NN (with a non-deterministic uncertainty) approved by a regulatory body? (automotive, medical, etc.)

Examples that jump to mind are Tesla's stereovision sub-systems (I am assuming NN are used), or NN based medical imaging classifier software (NN instead of Clustering or SVM techniques)

Here is a (longish) post I wrote about this issue (mainly from the perspective of verifying autonomous vehicles):

https://blog.foretellix.com/2017/07/06/where-machine-learnin...

Thank you very much on point. Thanks for sharing!

DARPA has acknowledged this problem and is working on it.[1]

See slide 11 for some of the technical approaches to making AI more interpretable.

[1]https://www.darpa.mil/attachments/XAIProgramUpdate.pdf

Various model compression schemes get rid of high-frequency noise in the model, combating over-fitting. This results in robustness against adversarial inputs.

Some of these schemes are motivated by rather robust theory. Gaussian Processes are a good starting point to learn more.

This is certainly out of my depth but how would this model 'reduction' improve provability?

It would (I assume) reduce instances of non-desirable behaviour. But how would it improve the evidence I am able to provide to a regulating authority?

To my layman's understanding; by reducing instances of undefined behavior as those instances should come from the noise of the data. Noise looks like structure in very-high dimensional views of data, so you need bounds on that stuff.

For autonomous vehicles the 'proof' to the regulator can be the signature of the engineer. - Fortunately some engineers have higher standards. - I think in the future the trainig data will be from simulations that create curriculum learning datasets so that the noise characteristics are perfectly known. The ML algorithms can be written with dependent types, so that you can prove your code does what you think it does.

Another challenge is inductive bias, which is a lot like confirmation bias. This bias comes from choosing an ML algo which is sensitive to certain information and blind to others. You need to navigate the set of all possible functions, AKA Hilbert space, to overcome it. Fortunately only a small corner of this space is relevant to our universe and Tensor Networks seem to address this problem.

It looks like a lot of work to put these pieces together but at least it looks like the problem is tractable.

Reply on news.ycombinator.com