Hacker News Reader

Comment by torbjorn | Hacker News Reader

I don't mean to come off as dismissive, and yes my comment is a sassy one. Machine Vision systems are proof that neural networks and other machine learning techniques are revolutionary. I really believe that.

But the notion of "Explainable AI" just reeks of upper middle management folks performing what they see as visionary risk management. When in reality these people don't know what they are talking about. Explaining exactly what's happening inside "the AI" is the domain of linear algebra. We can't be as exact in plain english but the generalization "it's fitting the training data" speaks volumes more than any _just_around_the_corner_ "explainable AI" secret sauce algorithm ever will.

View on HN

But the notion of "Explainable AI" just reeks of upper middle management folks performing what they see as visionary risk management.

AI says: "kill this person".

If it's only "upper management folks" asking "why" then I'm all about that visionary risk management!

Hell, I debug machine learning models every day, and most of my time is spent around working out "why". And the answer "it's just fitting the training data" is often wrong, insufficient, or misleading.

For example, I'm currently working on a sentiment-related task, and there's a particular class of misclassification where the model isn't taking the context of a word into account sufficiently.

In this case, the data is fit nicely, but the features it is fitting to don't contain the context. By adding tri-grams I can capture the context sufficiently to avoid this problem.

I can't speak for the middle management, but as I scientist I am very uninterested in AI based on neural nets because of the lack of explication. Success at prediction without a commensurate advancement in theory is pretty useless for me, because I haven't learned anything new about the world. And while Google et al are good at stacking ML models, those models are nowhere near as elegant and composable as scientific advancements.

You are right there is a large gulf between the deep learning theory hype and the measurable improvements in physical quality of life that the hard sciences have proved themselves capable of.

However I submit a notable exception! The vision system of self driving cars is a neural network that is currently changing the world and there is no lack of explication, we know exactly how it works. These "convolutional neural network" infer object boundaries from differences in intensity among matrices of pixels by scanning different patches of the pixel matrix and cross comparing these patches.

But you are right there is a lot of hype. I think part of the issue is the collective idolization of "algorithms". The limiting factor is data. And we don't have the data required to development models that are agent like in structure. Self interested agent systems, that employ subject-object language, have only been developed once. And they needed a training dataset measured in millions of years of evolution.

I think you're glossing over many accomplishments in ML. It's not just vision that is successful. There's NLP, optimization, ranking, voice, speech, recommendations and many other tasks that work well.

You say the limiting factor is data. But there has been a trend in the last 2 years to run ML on data generated from simulations (games, auto, chemical bonds, robots in VR, AlphaGo, etc). Simulations are dynamic datasets with unlimited flexibility. The better we learn to simulate, the closer we will get to reasoning. Both simulations and reasoning are based on object-relation graphs for describing the scene.

My vision is that we will build better, more precise simulators that would allow an AI to input a problem and run experiments and search ("what happens if"). Just like AlphaGo, but for the real world. A combination of neural nets for "intuition" and simulators for precision would solve the problem.

Simulators could be of many kinds: chemical molecule simulator, cell simulator, physical simulator, city traffic simulator, car simulator, flight simulator, and so on. Basically what's been the object of activity of supercomputing, but this time with neural nets selecting the experiments.

As a pseudo-engineer, I am somewhat worried about modern AI because you can have no idea when or how it will fail.

> as I scientist I am very uninterested in AI based on neural nets because of the lack of explication

Neural nets are more like reflexes than reasoning. Most of them are feed forward and some don't even have memory and can't solve references. So it's unfair to expect a job that is best done based on graphs or on memory-attention to be done by a rudimentary system that only knows to map X to y.

But they are not totally unexplainable - you can get gradients on the data and see what parts of the input data most influenced the output, then you can perturb the inputs to see how the output would change.

Reply on news.ycombinator.com