Comment by kylehotchkiss | Hacker News Reader

Comment by kylehotchkiss | Hacker News Reader

I'm not going to pretend to be especially knowledgeable in AI/Machine Learning/Neural Nets but I read something a few years ago that helped my perception of the limits of that tech: AI is not capable of explaining why it makes a decision that it does. This was a big flag to me in how much I trust things like full self driving - a car is making decisions with a trained cloud of intelligence that can't explain itself. I'm happy to see scientists and mathematicians trying to find the limits of what the tech can do, it's interesting to model it after how a human mind works, but we don't even understand human minds well enough to try to replicate how they learn intelligence. The world is closer to continuous numbers not discrete numbers etc. AI is still interesting enough to keep building on, Github's Copilot has saved me time just yesterday with very helpful autocompletes on some refactoring I was doing. More of that please - but let me be in the captains seat, deciding whether I want to accept the AI's decision or not.

I wish companies like Tesla would accept this and shift some of the self driving budget to better battery tech. That's the need of the hour with fuel prices rising like they are and global warming looming over us.

> AI is not capable of explaining why it makes a decision that it does.

So, yes this is true, but also not the full story. At this point we don't have neural networks that are also capable of explaining their reasoning, but what we CAN do is do a lot of introspection with that network. There is an entire field called AI Explainability that seeks to probe the network in various ways to help humans understand what is happening. Remember that you have total control over the network, and you can run inference thousands of times, or run pieces of the network, or feed test data in to the network.

I am a casual observer of the field but I see this "AI can't explain itself" thing thrown around a lot by people who don't know about the extensive research being done in explainability.

Also Tesla has a massive testing infrastructure that checks their network for regressions. So they will know if it suddenly starts failing in some area before they release it. Obviously this is new and complex tech so it is not perfect.

But I think self driving is important for their business, and they are probably investing heavily in both batteries and AI. And fully self driving electric taxis could eliminate the need for many people to own an ICE car at all.

>> AI is not capable of explaining why it makes a decision that it does.

That's not true for "AI". It's true for a particular kind of AI system, which are collectively known as "black box" approaches. Deep neural nets for example, are a "black box" approach because they can't explain their decisions, as you say.

There are other AI approachs besides deep learning. Recently, a system based on Inductive Logic Programming, a form of machine learning for logic programs, beat 8 human champions in the card game of Bridge:

https://www.theguardian.com/technology/2022/mar/29/artificia...

In Bridge, players must be able to explain their plays to their opponent, and the AI Bridge player in the article above, Nook, was specifically designed to have this ability _and_ play better than human champions.

Btw, lest this is perpetually misunderstood:

AI ⊆ machine learning ⊆ neural networks ⊆ deep learning

Think of it like human intuition. We haven't developed left-brain logic AI yet, but we are getting closer to training AI to outpace a trained human. They can look at a picture and say "That's a dog". That's an intuitive thought, not a logical one. A car could say "this situation feels dangerous", and ask the drive to take over, or it could just react and steer intuitively. It might not be able to reflect on its actions yet, but that doesn't mean it couldn't learn to drive.

Not that I think we WILL get to full self driving any time soon, but the car not being able to explain itself doesn't prevent us from getting there.

I don't think anyone does video to steering wheel in one network. But they do create a 3d model of the environment from sensors and do planning on top of that. So there is the explanation - what the car thought it saw.

You're right for self driving cars of course. Have you seen this paper? I have not had a chance to read in detail yet, but it seems to be looking at direct steering control from video with a NN:

https://arxiv.org/abs/2106.08314

Being able to "explain oneself" is a poor metric. Neural networks are not simulated human reasoning systems, they don't work in a way that is parallel to how we would see and explain something. Asking for an explanation is asking for a different kind of reasoning system. Every once in a while there is a "no such thing as tacit knowledge" article - to me this is the dual.

Anyway, IMO is what's more important is demonstrating an AI system can be trusted for the task it's doing (for example by being clear that the data it's predicting on is similar to what it has encountered before), and using the predictions responsibly (like not making a final high value decision based solely on model output). These things are not trivial and areas of ongoing research. But I think they will be more fruitful for developing useful AI vs trying to "explain" something in a way that will satisfy a human checker.

(Also, incidentally, there is work showing that much of explainability - for example feature maps - is just building something that produces output a human wants to see. It doesn't actually or necessarily correspond to how the NN calculated it output)

Humans are also not able to explain why they do what they do. Yet they are somehow able to drive cars.

The point is that they can explain why they are taking a left turn!

Can they?

How exactly does your brain do route planning?

How do you pick which lane to turn down in the parking lot when looking for a space?

Why did you get off at exit 14?

Wait, this isn’t even your turn, why did you go left here?

I mean, when you get down to it, why are you even driving to this deadend job?

But yeah, sure, humans can ‘explain’ their behavior, which is why we can trust them.

Well humans can explain critical aspect of what they doing while teaching other humans, furthermore they can improve explaining while teaching to be more effective in real time in current context, such as using analogies, stories, etc.

>> How exactly does your brain do route planning?

That is not the kind of explanation that is needed in AI systems. When people talk about "explainable AI", they literally just mean systems that can answer the kind of question that a human would be able to answer.

That's because a question that a human cannot answer is very likely to have an answer that a human will either not be able to understand, or will have to work very hard to understand... which is no better than no explanation

I think what people are looking for in ‘explainable AI’ is: when the AI makes a bad decision, they want to be able to look into the neural network and say ‘there: that neuron being set to that value is what made the AI mistake the cyclist for a drop kerb’. Then we can fix the value and the AI will not make that mistake again.

But when an AI gets sufficiently complex of course there won’t be explanations that make sense for those kinds of errors, because just like a human the AI is integrating lots of different bits of information that it has learned are important and it has limited capacity and sometimes it just gets its attention focused on the wrong thing and it just didn’t see the guy, okay?

Demanding that AI be explainable is fundamentally demanding that it not be intelligent.

Just to be clear, a neural network is not "an AI". "AI" is the name of the research field. We don't have "AIs" as in Science Fiction yet, and neural networks don't do anything "just like a human". When people in AI research talk about "attention" in neural networks, that's just an anthropomorphic, and quite unfortunate, name for a specific technique in training neural nets. It doesn't mean that a machine vision system has the ability to focus its attention in the same way that humans do.

That out of the way, there are AI approaches that can explain their actions just fine without going dumb. For example, I posted this comment earlier:

https://news.ycombinator.com/item?id=30872400

about an AI system called Nook that recently won a tournament against 8 human champions of the card game Bridge. In Bridge, players must be able to explain their moves, so an AI player without the ability to explain its decisions can't play a full game of Bridge.

> AI is not capable of explaining why it makes a decision that it does

There is a whole field of "explainable AI" - presumably in contrast with (the very popular) "opaque, inexplicable AI."

Personally I like decision trees, because you can trace the reasoning.

> AI is not capable of explaining why it makes a decision that it does. I know it's different, we also don't always know why make certain choices. We just pick one and say something to reassure ourselves. "I have a good feeling". A real AI will need to this type of "intuition".

> a car is making decisions with a trained cloud of intelligence that can't explain itself

Actually the AI is mostly just for object recognition, and the navigation and decision making are simple rules-based systems.

> AI can't explain itself

Has that been proven to be a fundamental limitation yet?

Reply on news.ycombinator.com