For AI to thrive, it must explain itself | Hacker News Reader

For AI to thrive, it must explain itself

economist.com

111 pointsjkuria65 comments

I think most people agree in the deep learning community that ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it.

In many cases, it depends on the context. For self driving cars, most companies only use neural networks for components of the cars and an explainable algorithm on top of that that interprets data from the components. If there was a crash, they can tie it down to "the neural network parsing the LIDAR did not classify the stop sign correctly, so the driving algorithm did not stop" without needing much interpretability of the neural network directly. If it's an end-to-end system, such as the nvidia car which connects the camera inputs directly to a wheel output, then explainability of that network is much more important.

There was a poll I saw on twitter a while ago (can't seem to find now) that had asked if users would prefer to be treated by an AI doctor with low explainability but 99% accuracy, or high explainability but only 70% accuracy. I think context is key here as well. If the expert doctor working in tandem with the explainable system is better than the non-explainable system, you'd want the explainable system. But if the non-explainable system is better even than experts, I think most people would want the non-explainable system.

> explainability is necessary to use it.

For that matter, freedom from daily blue screens and reboots is not needed to use an operating system.

All sorts of non-AI software has confusing states and quirks, yet is in wide use.

>But if the non-explainable system is better even than experts, I think most people would want the non-explainable system.

Maybe? You'd have to convince me in areas as medicine. Why are you recommending this treatment path? "Dunno. The computers says so."

Mind you, the doctor may well not in reality have a lot more solid basis for his recommendation. But at least there's the appearance of logical justification.

Is that because you care about verifiability though, or actually interpretability? If the system provides you a diagnosis without context, but later can be confirmed by actually testing for that condition, do you still refuse treatment because you don't know how the original diagnosis was suggested?

Oh, if it's a recommendation for running some tests, I can't imagine much pushback unless they're particularly invasive or expensive. It's not like doctors don't order large panels of tests at the drop of a hat.

If you then have treatment based on those tests, that's really a doctor being assisted by an AI at that point.

The current explanation for anything in medicine is "Dunno, we tested it and it seems to work". It is a much better one than "The computer says so", but it's one the computer can tell you too.

There is a huge swathe of problems where AI can provide tons of value, and where most of the benefits and costs of getting it wrong lies with the person deciding to trust the AI in the first place.

Your AI self driving car doesn't need to explain itself, it just needs to arrive safely. An AI that merely supports a human decision maker also might not, depending on how its applied. But the AI that might deny your access to treatment for a disease, reject your mortgage application or recommend to a patrolling police officer that he should pull you over, and not the next car -- those AIs should explain themselves.

If a self driving car decides to ram into a tree/another car at full speed regularly, im guessing there will be a huge demand for explainability, not just a collective shrug and "I guess we need to train it more dunno lol".

Yes, but at least the person bearing the cost is the same that decided to let the AI drive. That doesn't mean what you describe is desirable, but the fall out is rather contained to people who have consented to using the AI. Not so with the examples I listed.

but you are wrong, there are people who did not consent to using AI being injured in those situations, same as the police/sentencing example you brought up

What if the AI drove the car into a pedestrian? The pedestrian didn't consent to letting the AI drive.

It's a spectrum, not black and white. The self driving car is a lot closer to one end, and the other examples are closer to the other. None are perfect examples of the extremities.

What if a drunk drives the car into a pedestrian? Not much content there, either.

> What if a drunk drives the car into a pedestrian?

This is easy to explain. The driver is at fault. Explainability is what we are debating.

So the AI is at fault then.

the question becomes what can you >do< about it if you have no understanding of the logic it operates on.

Very much not.

Becauae there’s nothing to explain. The software is broken and needs to be fixed, and if the problem is severe and the manufacturer knowingly tries to cover it up, that manufacturer is culpable of negligence.

The software has no more agency than a cruise control.

Good luck explaining that to a judge who demands to know why the AI made the decisions that caused an accident, and what exactly the manufacturer is doing to prevent it from happening again.

There are tons of physical processes that aren't explainable. Most current petrochemical plants have inner workings that are only half known, at best. What are we doing about it? Modeling the failure, and engineering a solution that no longer fails in that situation. Unit testing our way to robustness!

> Most current petrochemical plants have inner workings that are only half known, at best. What are we doing about it?

When these processes kill people we very much do somethint about them.

> Becauae there’s nothing to explain. The software is broken and needs to be fixed

The second statement kinda denies the first. If broken software causes a severe or expensive problem, people generally expect an explanation. That is what a post-mortem is generally for.

I think it could be useful to distinguish systems which are required to automatically explain every decision they recommend as a matter of routine, versus systems which are not, but still are required to support post-mortem investigation of serious failures.

I think part of the problem here is what 'explainable' means and to whom. A Deep learning or Data Analyst may be looking for or accept one kind of answer (or even be able to comprehend it), while a Medical Doctor is looking for something else to hang his hat on, and the layman has an even different (lower?) expectation of explanaition.

Now perhaps we could design another set of algorithms (NN generated?) that would present the right type of explanation to each. Frankly, I'm not very interested in the layman's explanation, because what they want to hear is so disconnected from what's happening that it often seems irrelevant. That said, the placebo effect is real so you want the doctor and patient convinced that they are doing the right thing... so maybe it's more important (edit to add) for actual medical outcomes?

>> ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it.

I have to be able to debug anything I work with; debugging is necessary.

>> "the neural network parsing the LIDAR did not classify the stop sign correctly, so the driving algorithm did not stop"

It's important to look into why the stop sign was mis-classified, even if the classifier is a subcomponent.

I would go further: AI in a system that is in charge of safety critical system needs to be understood perfectly by its engineers. Right now, the community is trying to get by with shrugging and saying "dunno, but look at the reliability". In the long run this is not going to be good enough. We will need tools to dissect trained systems and build complete explanations of why it works (or doesn't).

It seems to me that understanding these deep learning models will be very tough since the reason why ml was necessary in the first place was the sheer complexity of the solution being too much for generations of programmers to solve

Understanding an existing solution is very different from trying to find it. We get taught things at school that seem simple and obvious to us now, but were extremely hard to discover or invent.

I see these training approaches as a tool that makes machines search the space of potential solutions for certain types of problems more efficient than humans. The next logical step is to try and understand these solutions and improve upon them.

Consider adversarial input. Right now we cannot tell which classes of adverserial inputs can exist for a given NN. We can only try to find representative examples. If you had a good enough understanding of how the NN works internally, you have a chance to derive the full set of adversarial inputs or - maybe -prove their absence.

The poll is biased. There is no evidence that performance and explainability are at odds.

With our current models they definitely are at odds. There is a strong correlation between higher average performance and lower explainability when comparing machine learning approaches like neural nets, linear models, decision trees, etc for nearly all tasks.

I cannot explain how I form the sentences I speak. I still find speaking useful.

Some people are missing the point.

> "In particular, machine minds that cannot explain themselves, or whose detailed operation is beyond the realm of human language, pose a problem for criminal law."

It doesn't says that the AI is NOT useful, but that it can't be used for the liability that it creates.

> "Dr Datta feeds the system under test a range of input data and examines its output for dodgy, potentially harmful or discriminatory results." ... " If the randomisation of sex produces no change in the number of women offered jobs by the AI, but randomising weightlifting ability increases it (because some women now appear to have “male”abilities to lift weights), then it is clear that weightlifting ability itself, not an applicant’s sex, is affecting the hiring process."

This is the most interesting part of the article. As it shows that it's possible to test the system for unlawful decisions without actually understanding how it thinks.

It sounds like a useful test if you get a positive result for discrimination. If the test comes back negative, things are more complicated.

This isn't as simple as it seems because a supposed job requirement, while superficially neutral, may be unnecessary. For example, police departments used to have height requirements. But height requirements discriminate against women. Do you need to be tall to be a police officer? Maybe it's really about projecting authority? Okay, how do you measure that?

Making job requirements neutral is a rather fuzzy area. What is the job really about? Who decides?

Any variable that's not independent of gender can be used to (imperfectly) detect gender. It may not be easy to figure out if machine learning is using it for something real or just as a substitute gender-detector. Randomizing gender while leaving everything else the same isn't going to detect this.

AI models simply exhibit the behaviour of the data they are trained on. If you make a model that tries to copy an individual persons behaviours, is it the AI model that is at fault or the person it's modelling? Where should the liability lay?

the problem with that also is that humans have emergent higher level reasoning, as a trivial, super simplified example, imagine if you trained a fictional AI on policy towards slaves before the worldwide ban on slavery, what sort of things would it suggest, its highly unlikely "banning slavery" would be one of them.

Here's another example: an AI, perhaps in a self-driving car, did something that caused harm to people and got the manufacturer hauled into court, and the judge issues a court order to "make it stop doing that."

How can the manufacturer comply, if they don't understand why the AI makes that decision, and how to change its decision-making process? This isn't just about accountability, it's also about the manufacturers' inability to control the behavior of a black-box AI system.

If the manufacturer explains to the judge that they don't really know what's going on inside the AI, the judge won't be inclined to cut them a break.

On the other hand, can you actually make a human not repeat given action? Or explain it?

I don't think that that is really the Problem. We also know how to teach people something, without knowing how our Human Brain works. The Problem is Companies can select for certain unlawful behaviour, but nobody can proof that later.

I’d like to simply know “how good is AI right now” at any given time. This information seems largely hidden.

For example, can any AI expert tell me: what is the exact date when Target corporation crossed the threshold of being able to face match 20% of its customers across different stores with 20% accuracy? How about 50% (for both of the different numbers, to keep it simple). How about 80%? 90%?

That’s just one example. If you’re tempted to reply with insights about face recognition, that’s not the point. The point is more that these numbers for ANY real world AI task (not just face recognition) are generally not shared, much less explained in the real world, even if would be possible to do so in an academic setting.

These are published in papers, but for real-world products, they're competitive advantage. Why should we share them with anyone other than our customers?

Data for very specific narrow tasks like face recognition is published. How the capabilities are combined in a special sauce, not so much.

So yeah, going off what you said, all these people saying “don’t worry, nothing to worry about with AI” haven’t the slightest clue what AI is up to outside of whatever bubble they are in.

>This means that even the designer of a neural network cannot know, once that network has been trained, exactly how it is doing what it does.

This is just false. The reason neural networks seem so mysterious is because in a fully connected neural network a large portion of the interactions (non zero w_ij in the weight matrix) are completely spurious. We look at the fully connected network topology and we throw our hands up. We can apply algorithms to trim out the spurious weights and what we’re left with is a logical circuit that we can analyze. Show an electrical engineer the circuit diagram of a 3 bit added and she’ll know exactly the function. Add a bunch of spurious circuits to that same diagram and of course it’s not going to make sense.

If those elements of the graph are "spurious", why not remove them from the very beginning? Or, should I say, not have them there?

Because we dont know which ones are spurious and the parent is pretty optimistic about the idea of being able to analyize which ones are/are not.

We only know which ones are spurious if we run the trimming algorithms. I wrote a paper about 10 years ago showing how to use generic algorithms on the weight matrix to remove spurious interactions.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/

Genetic algorithms?

Yes, sorry. Typo.

Do we know which ones are absolutely spurious? Or do we just know which ones are spurious with respect to a particular result that we got?

Like for this image of Grandma, these cells are recognizing it, so the others are spurious. Oops, for this other image of Grandma, a different subset are recognizing it.

If, say, the same 90% of the structure is not involved in recognizing any instances, then why wouldn't we have made the whole thing 10X smaller in the first place and trained that much smaller network? Or why don't we make it 10X smaller now? Trim away the dead weight and make it lighter and faster.

That’s what the genetic algorithm is for. You essentially add a entropy function that trims connections. If the performance is neutral or improves then the leaner network topology is subject to positive selection in a population.

If removing all of the spurious circuits is “cutting the bullshit” then what is left over is the truth.

Distilling a problem to its essence. Isn’t that the hardest step in explaining it?

Great article. I found a library that works with Keras last weekend; it shows which input features most strongly contribute to classifications. My little experiment: http://blog.markwatson.com/2018/02/trying-out-integrated-var...

Just as with self-driving cars, you have to look at the counterfactual. We don't have an interpretable audit trail for the decisions made by human employees either.

So under the status quo, what happens in the case of an aberrant or problematic decision and how does society cope with this lack of interpretability?

We try to piece together a plausible, retrospective narrative by looking at fact patterns and taking into account the education, experience, actions, explanations, and rationalizations of other humans. (Trusting these accounts is further complicated by human traits such as self-interest, emotion, and cognitive biases.)

There exist entire professional specialties devoted to this problem (litigation, internal investigations, police detectives, accident boards) who spend a lot of time on this. Even so, much of the time we still don't really know why exactly people made the decisions they did.

This inconvenient fact does not stop us from employing humans, and society has not collapsed under the weight of the liability issues.

It would be an interesting breakthrough if Alpha Go could explain its reasoning for various plays.

Maybe one day we'll write a deep net that captions other deep nets. Who will build the supervised learning training set though?

Since alphago plays the game differently (optimizing on odds to win, even if by a little, instead of the human method which involves intuition about shapes), it may be that its explanation if it were programmed to provide one would not provide any human readable 'insight' which an expert could then take to improve his or her game. Imagine a simpler example, some optimization solution using multiple dimensions and linear algebra. A description of how this works would be unreadable to most people. Since most readable language operates in the world of at most four dimensions without introducing jargon, this is the real problem, not that the program cannot explain itself.

At the higher level it isn’t just shapes. The human player uses the shapes the help estimate the future value of any group of stones. They are running a priority queue of likelihood’s for each part of the board. When they resign that probability has dropped to zero.

If you’ve played against a dan player a few times, you might have had the experience of them ignoring a move you made and play elsewhere. That’s what happened. It’s also probably the moment you lost the game...

I've played against dan players who describe this decision (tenuki, like the tenuki suit in mario, or "to deceptively play elsewhere") as mainly an intuition about "balance" which they somewhat vaguely describe as deciding that although the shape may not be good, the context of the shape is balanced, so it is OK to leave it be. According to Michael Redmond's commentary during the alphago matches (paraphrasing) often times the ability to calculate value is one of the last skills professionals actually master, far behind the concepts of shape and balance, and even among them some rely more on that skill than others. Lee Sedol, as an example, is relatively weak in this regard, Lee Chang-ho, on the other hand, seemed to be a prodigy just at this skill alone, allowing him to play a style which otherwise most observers called 'rigid'. So it is in fact common to find relatively high level amateur players who aren't good at calculating value.

Random amount of nodes with random connections assigned random weightings until something works best.

Not so sure it's possible for anyone to seriously claim to be able to explain the inner workings.

Biggest problem with these I can see is that there is more than one way to arrive at the given answer.

Simplistic analogy, let's say the equivalent nodes were represented by the value 20. Was this number a result of 5 * 4, 4 * 5, 10 * 2, 10 + 10 etc..

Right now, it doesn't matter, we're all just happy that 20 is correct and works for our use case. Maybe that will be good enough but I doubt it.

Counterpoint: explain, in detail, why you clicked the "comments" link on this post. What weights did you assign to all possible other actions you could have taken, and why, etc?

There's a pretty cool paper that visualises the different layers in a ConvNet trained on ImageNet: https://arxiv.org/pdf/1311.2901.pdf - page 4 has the layer imagery.

I don't think AI needs to be able to explain itself in order to be trusted. Human beings, in general, cannot give arbitrarily deep and valid explanations for their actions, and yet, somehow, we manage to come to trust them.

That's true, but in general humans are concerned about the consequences of their actions and are metacognitive about them, or at the very least can be punished for making the wrong choices, its an iterative corrective process.

How do you apply the same logic to a neural net? It has no comprehension of desirable outcomes, or what its choices actually entail and never can.

There is no way to tell a neural net that a human is not a signpost directly, so it knows in the future no humans are signposts. The only thing you can do is train it on a larger data set and hope it "gets it".

You can tell a human being that a human is not a signpost once, and they can abstract it to all situations.

Hell, you can't even tell them that a red octagon on a post with the letters S T O P on it is a stop sign.

Historically, observation and even usage precede understanding. Like planetary motion.

NNs (don't call it "AI", it's just one approach) are simply in a pre-scientific phase.

As are our own brains.

Sounds like robopsychologist might become a real job after all.

Humans don't explain themselves particularly well, yet they thrive.

'“the reason to give reasons is so that others will evaluate your actions and beliefs”. Today’s autonomous machines do not have their own interests to serve. Instead, their explanations are forged by and for human beings.'

This seems like the fundamental difference between AI and human intelligence at the moment. Our intelligence is based on our social prowess. We're always competing with, and exchanging ideas with our peers. AI should be modeled based on this.

Reply on news.ycombinator.com