Some AI Systems May Be Impossible to Compute
spectrum.ieee.org
spectrum.ieee.org
I wish companies like Tesla would accept this and shift some of the self driving budget to better battery tech. That's the need of the hour with fuel prices rising like they are and global warming looming over us.
How exactly does your brain do route planning?
How do you pick which lane to turn down in the parking lot when looking for a space?
Why did you get off at exit 14?
Wait, this isn’t even your turn, why did you go left here?
I mean, when you get down to it, why are you even driving to this deadend job?
But yeah, sure, humans can ‘explain’ their behavior, which is why we can trust them.
That is not the kind of explanation that is needed in AI systems. When people talk about "explainable AI", they literally just mean systems that can answer the kind of question that a human would be able to answer.
That's because a question that a human cannot answer is very likely to have an answer that a human will either not be able to understand, or will have to work very hard to understand... which is no better than no explanation
But when an AI gets sufficiently complex of course there won’t be explanations that make sense for those kinds of errors, because just like a human the AI is integrating lots of different bits of information that it has learned are important and it has limited capacity and sometimes it just gets its attention focused on the wrong thing and it just didn’t see the guy, okay?
Demanding that AI be explainable is fundamentally demanding that it not be intelligent.
That out of the way, there are AI approaches that can explain their actions just fine without going dumb. For example, I posted this comment earlier:
https://news.ycombinator.com/item?id=30872400
about an AI system called Nook that recently won a tournament against 8 human champions of the card game Bridge. In Bridge, players must be able to explain their moves, so an AI player without the ability to explain its decisions can't play a full game of Bridge.
Not that I think we WILL get to full self driving any time soon, but the car not being able to explain itself doesn't prevent us from getting there.
So, yes this is true, but also not the full story. At this point we don't have neural networks that are also capable of explaining their reasoning, but what we CAN do is do a lot of introspection with that network. There is an entire field called AI Explainability that seeks to probe the network in various ways to help humans understand what is happening. Remember that you have total control over the network, and you can run inference thousands of times, or run pieces of the network, or feed test data in to the network.
I am a casual observer of the field but I see this "AI can't explain itself" thing thrown around a lot by people who don't know about the extensive research being done in explainability.
Also Tesla has a massive testing infrastructure that checks their network for regressions. So they will know if it suddenly starts failing in some area before they release it. Obviously this is new and complex tech so it is not perfect.
But I think self driving is important for their business, and they are probably investing heavily in both batteries and AI. And fully self driving electric taxis could eliminate the need for many people to own an ICE car at all.
That's not true for "AI". It's true for a particular kind of AI system, which are collectively known as "black box" approaches. Deep neural nets for example, are a "black box" approach because they can't explain their decisions, as you say.
There are other AI approachs besides deep learning. Recently, a system based on Inductive Logic Programming, a form of machine learning for logic programs, beat 8 human champions in the card game of Bridge:
https://www.theguardian.com/technology/2022/mar/29/artificia...
In Bridge, players must be able to explain their plays to their opponent, and the AI Bridge player in the article above, Nook, was specifically designed to have this ability _and_ play better than human champions.
Btw, lest this is perpetually misunderstood:
AI ⊆ machine learning ⊆ neural networks ⊆ deep learning
There is a whole field of "explainable AI" - presumably in contrast with (the very popular) "opaque, inexplicable AI."
Personally I like decision trees, because you can trace the reasoning.
Actually the AI is mostly just for object recognition, and the navigation and decision making are simple rules-based systems.
Anyway, IMO is what's more important is demonstrating an AI system can be trusted for the task it's doing (for example by being clear that the data it's predicting on is similar to what it has encountered before), and using the predictions responsibly (like not making a final high value decision based solely on model output). These things are not trivial and areas of ongoing research. But I think they will be more fruitful for developing useful AI vs trying to "explain" something in a way that will satisfy a human checker.
(Also, incidentally, there is work showing that much of explainability - for example feature maps - is just building something that produces output a human wants to see. It doesn't actually or necessarily correspond to how the NN calculated it output)
Has that been proven to be a fundamental limitation yet?
> We are saying that there might be a recipe for the cake, but regardless of the mixers you have available, you may not be able to make the desired cake. Moreover, when you try to make the cake with your mixer in the kitchen, you will end up with a completely different cake.”
> In addition, to continue the analogy, “it can even be the case that you cannot tell whether the cake is incorrect until you try it, and then it is too late,” Colbrook says. “There are, however, certain cases when your mixer is sufficient to make the cake you want, or at least a good approximation of that cake.”
The idea is that a neural net is a kind of data structure used in AI, like a decision tree or a decision list (like a decision tree but it's a list). There are different algorithms that can "compute", i.e. construct, a decision tree from data. In modern parlance we say that the decision tree is "trained". Same goes for neural nets, except the network itself is typically constructed beforehand, and manually (we refer to it as the "architecture" of the neural net) and only its weights need to be tweaked until it has a good accuracy- at which point we say the training algorithm has "converged".
It's all a bit confusing because in common parlance there is little distinction made between a neural net's network (its architecture), the algorithm that trains the neural net by finding the weights that minimise its error (backpropagation) and the neural net with trained weights (the "model"). Sometimes I wonder if this distinction is clear in the minds of people who actually train those things.
Btw, the study is solid and meaningful. It's a theoretical result. More of those are needed in machine learning, we got plenty of empirical results.
I am pretty hip to modern AI workings, and that is what I assumed they meant by compute. What I know also tells me that "compute" as a concept is lacking for what we might later call holistic AI, and if, for instance, they are conjecturing strictly SGD proved networks, then I maintain my position that they are merely noodling, with AI and terminology alike.
Intelligence is not entirely computable; nor is occurrence entirely probabilistic. Respectively, there is randomness at work, and dynamic systems that produce novelty.
In my experience, it's not the alteration that creates an unpredictable outcome; it's the lack of nuance. Accuracy requires an insane amount of details, and it appears that some people might be waking up to that fact that the brain is better suited to handle those nuances.
I work with NLP and have discovered advantages in leaving some decisions up to the person using AI rather than the computer. One advantage being accuracy :-).