Is artificial intelligence permanently inscrutable?
nautil.us
nautil.us
A super casual modern day example would be an assertion like "Joe must be gay". You never know until you ask him, and that is if he's honest with you. But everyday-life is full of gross approximations based on bias and partial information.
But the key, even in Joe's case, is you have to go back to the real source. If all we can do is talk about Joe, we can never know for sure. Much like celebrities we are so certain are gay, but haven't come out openly yet. So as long as computers can't leave their data centers, there really isn't any amount of data that can prove anything to them. They can't experience the truth, because ultimately the truth rests on experience and experience of evidence.
Show some child a cat and ask why it's not a crocodile. They will be able to explain it through the differences in shape, color, behavior and other features of those animals. Whether you consider it post-hoc is unimportant. The explanation is still real and relevant.
So people can explain their deliberate reasoning. Also, reason about reasoning. If they truly can't, it's called gut feeling and no one rational trusts those for solving complex problems with important outcomes. Especially at scale.
Neural networks, at least right now, have no capabilities of this sort. There are some attempts at visualization, but they are inherently limited, because they must make assumptions about the domain of the problem.
You may be able to explain why a cat is different than a dog, but your brain doesn't go through that categorization process before recognizing a cat.
Sometimes I "know" a design is bad. Explaining why to others requires stepping through a process that May or may not be the one my brain used to make that approach "feel" wrong.
Take for example persons with androgynous features and the difficulty some people have with identifying them. Children are particularly useful for this insight since they are unabashedly curious and inquisitive, and it's not uncommon to hear a child blurt out "is that a boy or a girl?"; it would seem to me that their categorization process, even if not refined, gets goofed by androgyny; something as simple as hair length or facial hair greatly influences a child's ability to easily discern if persno is male or female.
At some level there is a categorization process that seems to happen with humans when it comes to recognition, we just assign high confidence to certain factors. In the case of children, it seems to be they look at person and use common factors like hair length, facial hair, and body shape to determine gender. The sound of one's voice also helps, but this usually isn't something we can pick up as easily or from afar in public places. As an adult, we probably have more refined points, but they are essentially the same as a child's point of view. Instead of "body shape" adults look for specific facial features, size of breasts, walking gait, style of clothing, hair style, etc. These are all just granular differences of how a child understands it.
This gets into personal experience, but when I was in college, I had long hair and really fine skin/facial features. All the time I would hear little kids ask "why does he look like a girl?" or "why does that girl have a beard?". I had no issue with their confusion, they were just curious kids. Something confused their understanding, and once it was explained "sometimes men have long hair", they just sort of accepted it.
We've added instrumentation to our recognition capabilities in order to socialize and gossip about derived categories through language. In addition, we (can sometimes) use the formalized expressions of these categories prepared for socialization to ensure internal consistency.
Artificial neural networks that did something similar could have a number of major philosophical and practical advantages: - enable decentralization of data sets - grow with humans rather than apart from them - enable identifying problematic internal inconsistencies
The examples I am more worried about are not knowledge based (or judgement based). The ones I am worried about are the stories we tell about the _actions_ we take, why we choose A instead of B. Answering these requires us to introspect our own behavior and that is exceptionally difficult. Take for example a question like "how did you know it was safe to cross the street?" Many people will tell you that they 'looked both ways and saw nothing was coming' but if you have them wear earplugs they will get run over because it was really the fact that they were listening for the cars. Or say you meet someone and you decide you don't like them, you can't really go back an undo that, but it might actually have been the case that you didn't like the perfume they were wearing because it was the same one that your ex used to wear, and it had nothing to do with the person themselves. Virtually no one can catch all the cases like this because we are trapped by our own brains and our own blind spots. Buying your brain's bullshit without attempting some independent verification (eg by asking someone for another perspective or doing little experiments on yourself) is a exceptionally common, and difficult problem to overcome.
One example of this is Anton-Babinski syndrome [1]. The patient can't see but is completely sure he can.
[1] https://en.wikipedia.org/wiki/Anton%E2%80%93Babinski_syndrom...
Humans have a very hard time uncovering so-called "lurking variables" [2] and identifying such paradoxes. I don't see how a neural network (or other machine learning tool) could do so on their own, but I don't know that much about machine learning. So, I guess I have two questions for the experts out there:
* If all training data is affected by a confounding variable, can a machine learning algorithm identify its existence, or is it limited by only knowing a tainted world?
* Once we have identified such lopsided data and understood its cause, how do you feed that back into your algorithm to correct for it?
---
[1] https://en.wikipedia.org/wiki/Simpson%27s_paradox
[1a] https://en.wikipedia.org/wiki/Simpson%27s_paradox#Kidney_sto...
In cases where it's very likely the doctor would make a different decision, it should flag it for human review.
This is tackled in the recently popular field of study called 'counterfactual inference'.
Once you start to trim out the spurious connections you start to see that you are left with a logic design with integration/threshold circuits instead of straight binary circuits that we're used to seeing. There are even certain universal network patterns what will emerge to perform different functions just like in binary circuit design.
I wrote a paper about this in 2008 that's now been cited about 150 times. It's using Artificial Gene Regulatory Networks instead of Artificial Neural Networks, but the math is the same and the principle still holds:
Just like an International Master commentator can explain most of the moves of a Super GM, so can an interpretable simple model explain the predictions of a very complex black box model.
The work by Caruana referenced in this article actually culminated in a method to get both very accurate models and still retain interpretability.
http://www.cs.cornell.edu/~yinlou/projects/gam/
More recently there was LIME:
https://homes.cs.washington.edu/~marcotcr/blog/lime/
And there are workshops:
http://www.blackboxworkshop.org/pdf/Turner2015_MES.pdf
We will get there. 'Permanent' is a very long time and in the grand scale of things, deep learning is relatively new.
Critically though, I will say something to the effect of "but if you try and break the net open and see how this specific net came to it's result, it will look like spaghetti"
So it's a roundabout way of saying "junk in; junk out." That holds true for any learning system, including human animals. The thought process of humans is inscrutable thus far, and I think that future computing will be similarly inscrutable if we do it correctly.
See [1] for a discussion of both.
[1] https://blog.foretellix.com/2016/08/31/machine-learning-veri...
> As exciting as their performance gains have been, though, there’s a troubling fact about modern neural networks: Nobody knows quite how they work. And that means no one can predict when they might fail.
Nonsense! Cross validation. Develop hypotheses, develop subsets of data to prove or disprove given hypotheses, observe how the network reacts. All of these people complaining about not being able to understand what's going on are either reporters, bloggers, or machine learning dabblers looking to say something seemingly unconventional.
From your linked article, which gives more specifics as to the argument:
> Because ML systems are opaque, you cannot really reason about what they do.
Yes, it is possible to reason about a system even if it is "opaque"; the discipline is called reverse engineering. Or the scientific method.
> Also, you can’t do modular (as in module-by-module) verification.
You can do "modular verification" in a variety of ways. Start with analyzing the behavior of each layer and how that changes as you incorporate more layers. It's beyond the scope of this comment to go into it beyond surface level, but there are a lot of papers written about it, google "analyzing neural network hidden activations" or something.
> And you can never be sure what they’ll do about a situation never encountered before.
Humans can never be sure what they'll do about a situation they haven't encountered. Or engineers. They can simulate the events that they can think of, but we can also do that with a neural network.
> Finally, when fixing a bug (e.g. by adding the buggy situation + correct output to the learning set), you can never be sure (without a lot of testing) that the system has fixed “the full bug” and not just some manifestations of it.
Fixing the "full bug" is often not something that can be done in traditional software development "without a lot of testing". Machine learning works the same way.
If you want if/then statements, use a decision tree. If you want strong accuracy on predictions, use a neural network. It helps to know what you're doing when verifying results. You will run into trouble if you don't know what you're doing, as per common sense.
Nobody understands exactly how painkillers (aspirin, paracetamol etc.) work on the molecular level. Yet they are generally held to be quite useful.
Edit: electroconvulsive therapy is an even more extreme example; we have no clue how it works, but it's very effective on severe depression. We only know it works because some Italians back in the olden days, before ethics committees were invented, decided to electro-shock a bunch of "crazy people" to see what happened. The reason they tried it was that electroshocking of pigs for slaughter had been observed to give a temporary anaesthetic effect.
What I meant by "Because ML systems are opaque, you cannot really reason about what they do" (perhaps I was not being clear) was this: You can indeed _observe_ what they do, but being able to actually inspect the source makes verification much easier and more reliable: You know what parts of the logic you have covered, you can think of "danger areas" (and direct testing to them), and you can simply check whether all the cases you can think about have been covered in the source.
With opaque systems, you have no idea whether e.g. your ML-based autonomous vehicle will recognize people-painted-on-a-bus as people-on-the-road, until you actually test for that. And then you have to test people-painted-on-a-truck.
What I meant regarding "modular verification" is that you can check sub-modules according to some spec (or at least according to informal comments). This is quite different from what you can do when analyzing NN layers.
I suspect you are right in claiming (in your last paragraph) that one will always have to choose between "more understandable" and "more accurate". But I think we can do various things (the DARPA suggestion being one of them) to make the "more accurate" solution more verifiable.
The pneumonia project referenced in the article had Caruana voting against implementing the most accurate model: A neural net. Instead they went with a way less accurate logistic regression model, one they could safely implement in production, inspect, explain, and defend to the doctors.
Nobody does quite know why neural networks work so well. There is a Nobel Prize waiting there for someone or some team to solve this with mathematical (or physics) rigor.
Nodes in neural network layers can represent multiple features, or share feature representations. Do we know if a neural net (and which part) is targeting skin color, or acne? Do we know that credit risk models are targeting sex, even though we left out this feature (it may infer this from other features)?. Depending on the application, this is important to know for certain.
Reverse engineering sure does work, but can we fully find out the source code from a program, just by fuzzing inputs and looking at outputs? Or are we only looking at (perhaps a small part of) its behavior?
> You will run into trouble if you don't know what you're doing
Likewise: You will run into trouble, if you don't know for sure what your models are doing.
http://blogs.wsj.com/digits/2015/07/01/google-mistakenly-tag...
> as per common sense
https://en.wikipedia.org/wiki/Commonsense_reasoning#Commonse...
I think eventually, within a couple of decades, we will have AI that correlates well enough with human thought process, and has enough knowledge of the world, to be able to introspect and explain in various levels of detail, in natural language, images and other human readable constructs, why it has reached a certain conclusion. And we will be able to experimentally verify those explanations.
Eventually it might grow into a full-fledged science, but it will probably take an awful lot of time.
Where we pass into alchemy though is the interplay of these basic components with each other and the parameters they encounter while running, this is where complexity happens. Part of this lies in the very nature of the tasks we use them for: we basically push a cart of raw data in front of a set of "AI" solvers and expect them to do something with it. When that doesn't work, start over, tweak parameters, and try again.
I agree that there is no sufficiently useful intellectual framework for creating these artificially intelligent components, and that shows not only in the uneven success rates and performance, but also in the surprising fact that experts in very different AI systems can usually create components with similar performance characteristics for a given problem, despite using very disparate strategies.
This means that under many circumstances, you can build a model that satisfies certain abstract properties, test it, and have a high probability of generalizability. I.e. we've circumvented the "understanding" stage.
In fact, I'd say that the core of machine learning (as opposed to merely statistics) is exactly this.
Machine learning research is very empirical. With a philosophy of doing lots of experiments and tests, and discovering what methods work best. That's what science is. No scientific field starts off with complete knowledge and understanding, they have to do a lot of experiments to discover the general laws.
Some people dislike empiricism and want pure, provable math. Machine learning isn't a field of mathematics, at least not in practice. But that doesn't mean it's not a science.
So it is up to us to fill the gap in our understanding because that is what machine learning ultimately says about the subject. It tells us what we don't know. If we knew all about the subject, our predictions would match the predictions of the machine because there is only one reality we're both observing. But if there is any gap, then the machine is telling us what we don't know, not what it (of all things) knows. It's just crunching numbers. It doesn't "know" anything.
However, I think the human should not be in the loop. The network should have another semantic layer that serves communication. It can be done from the ground up like Steels or Vogt have been doing.
In other words, yes we need insight, but I prefer it through introspective networks. The network should be able to explain itself.
I don't think there has been much progress in his work though. :-) I asked him a few months ago if he saw some promising inroads to embodied cognition, but it didn't seem like it. Still it's a part of AI that's neglected on the moment.
PS: Of course there is research, just not mainstream. Der, Parisi, Tani, Bongard, Prokopenko.
But still, the results are very difficult to interpret. Yes you can verify that the equation works, that it predicts the data. But why does it work? Well who knows? No one can answer that. Understanding even simple math expressions can be quite difficult.
One biologist put his data into the program, and found, to his surprise, that it found a simple expression that almost perfectly explained one of the variables he was interested in. But he couldn't publish his result, because he couldn't understand it himself. You can't just publish a random equation with no explanation. What use is that?
I think the best method of understanding our models, is not going to come from making simpler models that we can compute by hand. Instead I think we should take advantage of our own neural networks. Try to train humans to predict what inputs, particularly in images, will activate a node in a neural network. We will learn that function ourselves, and then it's purpose will make sense to us.
There is a huge amount of effort put into making more accurate models, but much less into trying to interpret them. I think this is a huge mistake, because understanding a model lets you see it's weaknesses. The things that it can't learn, and the mistakes it makes.
At least we should have a standard for characterizing their accuracy or something like that ...
Similar to the human brain in that respect. Ironic that the human brain used itself without understanding how it works to eventually create something it uses without understanding how it works.
-college degree (Doctor)
-employee referral (engineer)
-FizzBuzz (engineer again)
-licensing exam score (driver)
-popular election (supreme commander of the armed forces)
E.g., AlphaGo is clearly good at Go. Yet it makes moves that can appear inscrutable to a human expert.
(NB: I'm not arguing that inscrutability is or is not an issue for AI systems, just suggesting that third-party validations don't meaningfully address the issue one way or the other.)
It's like trying to create an AI to play chess but the creators focus so hard on keeping it from making moves that result in its losing the queen that they forget that the objective of chess is to win.
Think Different! O'well.
in a shallow net, maybe college student selection based on sat scores gets a heavy weight/low threshold/whatever. in a shallow linear combo, this will likely always carry a large weight.
in a deeper net, it might be able to learn that SATS are a great predictor except for when X Y Z or some combination of those are some particular value, in which case it might be wholly irrelavent. The deeper it is, the longer it will take to train, but the more it can handle exception cases/trends and approximate reality
A standard example is for a face detection algorithm. The first layer will do edge detection, the next layer will combine edges into corners and simple shapes, the next layer will maybe use those shapes to look for features like eyes, noses, mouths, etc., and then the next layer will maybe combine those features to look for a whole face.
I wrote a more detailed answer here:
http://stats.stackexchange.com/questions/222883/why-are-neur...
The first layer might recognize and respond to pixels in particular parts of an image, the next layer will group certain of those pixels-responses together into an abstraction you might call a "line", the next layer will respond to certain groupings of lines and add a level of wiggle-room regarding where the lines are in the image, and the final layer will judge whether a combination of groupings constitutes the letter "A". Or at least, if you spent a bit of time poking at a deep network, giving it slightly different inputs, you might eventually conclude that this is what the layers were doing.
Without layers, you're basically just approximating a simple function or mapping with one level of abstraction.
The other responses to your question in this thread are as good I as could give you, but I would feel like I am recounting ideas that may be true but for which there is little evidence.
This seems analogous to 90% of (random, unreplicable) science these days.