Machine Learning Confronts the Elephant in the Room
quantamagazine.org
quantamagazine.org
What if an algorithm actually had a focused sight? Look at the brightest feature, blur the rest, then try to find another distinctive details, look closer into the parts of the image until the most of it becomes clear. Let other neural network control the scanning process and loop until it develops some confidence in the result. Can it work that way?
So the dramatic changes seen in the image seem to be artifacts of the visualization and not of the network's performance.
Modern networks are instantiated with random noise but end up pretty much the same as previously hardcoded filters.
https://www.nature.com/articles/s42003-018-0110-y
Of course the neurons in our brain work in a very different way, but at an architecture level the strategies of natural and artificial neural networks seem to match.
As the intro to the the article says (you now noticed, didn't you?), the way DNN vision systems are different is that "unlike humans, they can’t do a double take". And those "double takes" is what our vision pipeline does all the time.
> an ability humans have that AI lacks: the ability to understand when a scene is confusing and thus go back for a second glance.
Wouldn't the machine return a low confidence score when a scene is confusing? If not, why is this difficult to get around?
If so, why can't we just call this a computation difficulty problem, simply requiring a way to go back and spend more effort later when required? A problem like this would simply require better computers, and patience.
In other words, Is this problem as deeply rooted as the article suggests? Or is it simply a problem with the popular approach to machine vision?
* return a result
* return a list of results
* return a list of results with a confidence weightins
* return a list of results with multiple weightings (confidence and complexity)
how many if statements would you need to write for each if this was applied to a 3x3 grid like tic-tac-toe? how would you know you are correct?
"Confusing" would be a poor choice of word here though.
Neural nets should return a low confidence score. But, the popular approach (described below) ignores that. Neural nets ignore confidence because of a technique called softmax [1].
This happens as the final operation of a neural net, and is required for training.
Softmax is a tool to make an array of positive numbers look like a probability distribution:
out = x / x.sum()
But then in inference/prediction, the confidence is ignored. What if x.sum() is small? That would imply that the network is uncertain.
But you can use component wise normalization (sigmoid) instead of combined normalization (softmax). These correspond to the assumption that the classes are independent (component wise sigmoid) or mutually exclusive (softmax).
"and being wrong on a confident prediction gives huge penalties using the popular maximum likelyhood loss functions" - It should.
For classification tasks, the labels are usually encoded as a one hot vector (one in the position of the correct class output, zeros everywhere else). If you don't normalize outputs to be between zero and one, it becomes a regression task - you are essentially asking the model to fit your one hot encoded label. That's not desirable, because we don't care about the actual value of the output for the correct class. Whether it is 0.1, 1.1 or 1001 it is the correct output as long as it's larger than outputs for other classes. That's why we want to take the largest output, and scale it in a way that it's always less than one. Its distance from one depends on how much larger it is than other outputs (the confidence of the model in this prediction).
Without normalization, the model that outputs 1000 for the correct class and tiny values for all other classes would get severely penalized because the labels says it should be 1 in that position (so the error is 1000-1=999), even though the model made the correct prediction.
There's some confusion about this (e.g. https://news.ycombinator.com/item?id=18054447 ), so hopefully my explanation makes sense.
1. Transform the components to e^x. This allows the neural network to work with logarithmic probabilities, instead of ordinary probabilities. This turns the common operation of multiplying probabilities into addition, which is far more natural for the linear algebra based structure of neural networks.
2. Normalize their sum to 1, since that's the total probability we need.
One important consequence of this is that bayes' theorem is very natural to such a network, since it's just multiplication of probabilities normalized by the denominator.
The trivial case of a single layer network with softmax activation is equivalent to logistic regression.
The special case of two component softmax is equivalent to sigmoid activation, which is thus popular when there are only two classes. In multi class classification softmax is used if the classes are mutually exclusive and component-wise sigmoid is used if they are independent.
E.g, we have "regression" and "classification" instead of "supervised continuous prediction" and "supervised discrete prediction".
Plot max(0, x) and softmax(0, x) functions, and it should become clear.
Yeah it makes sense that this is a super important function, but I still feel like one could just remember the principle that "exponentiation followed by normalization is a smooth approximation to maxarg."
This opens up possibility of using something else than softmax in there.
If you just say "normalization layer" how am I supposed to know which normalization you're talking about?
In other words, if you only have two object classes, the magnitude of the outputs does not matter, the uncertainty is measured by the relative difference of the outputs.
The only way to measure the confidence of the model that the output is "cat OR dog", is to have another class (e.g. "chair"), only then, looking at all three outputs you can estimate the confidence of the model regarding "cat OR dog" predictions (vs 'NOT (cat OR dog)"). For example, if [cat, dog, chair] outputs are [0.03, 0.01, 0.05] then we know the model is not confident that it's either a cat or a dog, but if the outputs are [0.75, 0.25, 0.05], then it's clear it is.
Present neural nets have no "more effort" or "less effort" knob. For the same input they always produce the same output.
The article says a different thing: humans rerun the "algorithm" again, but this time with the knowledge that something was wrong in the previous run (lets say color), and this knowledge will tweak the way it runs again, maybe even running a completely different algorithm the second time (a network more specialized in color but which is worse at shapes)
Consider, can you recognize that distant relative you have never seen? Why not?
We need a cognitive model for vision... This is all leading back to David Marr, who wuda thunk.
Second, even many people are surprisingly bad at decoding incongruous scenes, which is why hidden object games are a thing.
Presumably the couch showed up because it rose from <0.5 to 0.57. They don't tell us what the score is for the chair, it might even be higher than the score for the couch, since their visualization clearly doesn't sort by probability.
By the way, something that is not described in the article- the referenced paper also describes experiments where objects where translated inside the source image itself - so not in another image. And the tested networks were still confused.
So it's not "foreign" objects that are the problem. It's that neural nets generalise very poorly. Show them something they haven't seen before and their ability to recognise it falls off a cliff.
Well, you could randomly generate and render 3D objects, and paste them into photos. This will give you a huge space of 'elephants'.
What you propose is, essentially, training on a simulation. That doesn't work very well because statistical machine learning algorithms generalise very poorly from training data to unseen data and when your training data is a simulation (such as a 3d model of an object) and the unseen data is the real world (such as a video still, or a digitial photograph of that object) the resulting model will be very bad at predicting the unseen data, so identifying objects in images of the real world.
It might seem to you (and to me!) that modern 3d simulations (as in high-end CGI and video games) is a very faithful simulation of the real world, but that's primarily because we look at modern 3d through human eyes and perceive it with our (presumably) human brains. We don't know how that works exactly, so we can't reproduce the process in computers, yet, therefore machine vision systems essentially "see" and "perceive" something completely different than what we do. In the end, what looks like a very close approximation to you, is pretty much useless for them- or in any case, they are not able to learn to generalise from that to real objects.
This really is not what I expected, being a layman.
Do you have any more data on this topic of training on a simulation?
https://ai.googleblog.com/2017/10/closing-simulation-to-real...
Simulating many years of robotic interaction is quite feasible with modern parallel computing, physics simulation, and rendering technology. Moreover, the resulting data comes with automatically-generated annotations, which is particularly important for tasks where success is hard to infer automatically. The challenge with simulated training is that even the best available simulators do not perfectly capture reality. Models trained purely on synthetic data fail to generalize to the real world, as there is a discrepancy between simulated and real environments, in terms of both visual and physical properties. In fact, the more we increase the fidelity of our simulations, the more effort we have to expend in order to build them, both in terms of implementing complex physical phenomena and in terms of creating the content (e.g., objects, backgrounds) to populate these simulations. This difficulty is compounded by the fact that powerful optimization methods based on deep learning are exceptionally proficient at exploiting simulator flaws: the more powerful the machine learning algorithm, the more likely it is to discover how to "cheat" the simulator to succeed in ways that are infeasible in the real world. The question then becomes: how can a robot utilize simulation to enable it to perform useful tasks in the real world?
The difficulty of transferring simulated experience into the real world is often called the "reality gap." The reality gap is a subtle but important discrepancy between reality and simulation that prevents simulated robotic experience from directly enabling effective real-world performance. Visual perception often constitutes the widest part of the reality gap: while simulated images continue to improve in fidelity, the peculiar and pathological regularities of synthetic pictures, and the wide, unpredictable diversity of real-world images, makes bridging the reality gap particularly difficult when the robot must use vision to perceive the world, as is the case for example in many manipulation tasks.
Note the bit about deep learning algorithms being very proficient "cheaters", which I missed in my comment above. Indeed, one way to fail to generalise from a simulation to the real world is to "overfit" to the defects in the simulation!
Like the linked blog post, most material you are likely to find online focus on training robots with (deep) Reinforcement Learning, I think because that just happens to be one domain where it is even harder to collect training data than good old supervised learning for image recognition. I can find virtually no source referring to the "reality gap" in the context of purely machine vision research- it's just not the done thing to train vision algorithms on simulated data, for the reasons described above, consequently it's very difficult to find hard data on why it's not done.
A good related source (with a bazillion references) is the following blog post, discussing the difficulties of deep RL, which is primarily trained on simulated environments:
https://www.alexirpan.com/2018/02/14/rl-hard.html
The post makes only passing referenct to the "reality gap" but it should give a good idea about the ins and outs of training in simulated environments.
https://rodneybrooks.com/forai-steps-toward-super-intelligen...
If they all agree you have a high confidence, if they mostly disagree you have a low confidence.
[1] https://en.m.wikipedia.org/wiki/Ensemble_averaging_(machine_...
To an extent it shows the gap between neural networks being able to classify things, and actually being able to understand what they are seeing, which will always limit their utility beyond classifying images that look like other images they've seen before.
I tried it on a couple of online demos, neither spotted that there was a car in the picture.
A space collision avoidance AI that identifies a road vehicle as a road vehicle and expects it to behave as one normally behaves would create problems. A human can look at a 1936 Ford truck floating in space and know it's not going to make a sudden left turn. An AI working in space would still treat it as debris if a human told it "that's a truck, it's not going to do anything different unless acted upon."
A system that can't recognise objects out of context is either completely useless, or only useful to the extent that it can recognise a very large number of objects in a very large number of varied contexts. Essentially, it's a system that learns primarily by memorising specific cases, rather than memorising and generalising to new cases.
This kind of system is limited by the number of objects and contexts with which it can be trained in practice. Which is why you need large datasets to train machine vision algorithms- because you basically have to show them many examples of everything you want them to learn.
In your example, the problem is not that the "AI" would not be able to recognise a car in space as a car, in space; it's that it would not be able to recognise it even as debris - unless it had already seen such "debris" before. Obviously, if it hadn't observed the behaviour of such debris it would also be incapable of predicting its behaviour, also.
The inability of statistical machine learning models to abstract and extrapolate from available observations, in the same way humans do, can be very surprising, especially in light of their othewise excellent abilities to recognise objects they have seen before.
This is analogous to a person from the early days of photography having no knowledge of the possibility of image doctoring, and us feeling "smug" because a photo like this would elicit confusion.
It seems to me you just have to run your NN on pairs of doctored and undoctored images, and create a separation of concern between scene continuity and object recognition. It's likely that a lot of implementations currently rely too much on surrounding context for successful categorization.
It's also worth exploring certain image analysis techniques during pre-processing to draw out certain attenuated aspects. Edge detection is par for the course, but things like shading are also important to attenuate so that the NN has an easier time learning.
If we're dealing with multiple frames and not just a single image, this opens up a whole dimension of temporal analysis that should make it trivial to separate the background from juxtaposed images.
This may fix this problem, but what other visual issues will we uncover that might be fixable with the right kind of training but which we haven't yet accounted for.
For example, we see so many optical illusions because of how our eyes and brain (and the neurons between them) work. These are often times caused by our sensation or perception incorrectly picking up data, but on purpose (well, as much purpose as an evolved system could have). For example, how neurons in the eye exaggerate differences in brightness or how we are overly sensitive to perceiving faces so that we see them even where they don't exist. What impact does the lack of these have on an artificial vision system? For starters, it would likely indicate the system wouldn't recognize the optical illusion unless trained for it, but are there any other impacts that we need to take into consideration, especially when considering having human lives depending upon the system?
The problem with videos and images vs. the real world is the fidelity, you can't trust images or videos that are previously captured through other equipment. This barrier wouldn't exist if the AI can "see" into the real world, the skin colors of animals can be trusted there!
Thanks for giving us ML researchers a TODO. We'll get to work right away.
Does such a human exist? Other than perhaps someone blind from birth who had their sight restored, I don't think you could fine such a person give how conditioned we are to recognize animals, perhaps even on a biological level. Closest I can think of is animals that are camouflaged, and I think we would see many humans making mistakes on those. I've definitely have, especially with regards to insects that look like plants. But those animals specifically evolved to throw off predator detection systems.
It would be similar to a situation where a human, who sees an improbable object for a given environment, reevaluates his initial perception/understanding of that environment. I can't come up with a good example of such a situation, but you get the idea.
The images generated here could be viewed as a variant of adversarial examples, in which small image perturbations (imperceptible to humans) cause a large shift in the network’s output. The images we generate are of a somewhat opposite flavor: while we do not limit the magnitude of the difference between the original and modified image, the detectors are sometimes “blind” to the inserted object. In addition, our examples are not “targeted” in the sense that no optimization process is required to generate them; they seem prevalent enough so that a simple scan of transplanting translated versions of one object in the other can give rise to multiple wrong interpretations.
How do I know I can do this? Simple. Every scene I look at I can basically translate or imagine that scene in my head as wireframe scene or some low poly scene as if it was generated by a computer. Similar to if I look at wireframe scene generated by a computer, my mind can translate it into a scene that looks real. Try it, you can do it.
Second, I can look at an actual low poly wireframe model of an elephant and associate it with the word 'elephant.' I do not need color, or detail to know it's an elephant. In fact, with just color and detail alone it is harder for me to identify an elephant. For example if someone takes many very closeup photographs of parts of an elephant like its eye, skin, ear, etc.. and asks me to guess the subject by interpreting the pictures... I become fully aware that I would be accessing a slower, different part of my brain to deduce the meaning. This is a stark contrast to the instantaneous word association established when I look at a wireframe model of an elephant. The speed difference between both ways of identifying an elephant indicate to me that geometric interpretation is the primary driver behind our visual analysis and details like color or texture are tertiary when it comes to the identification of an elephant. I believe the visual cortex determines shape first, then subsequently determines word from shape.
If you feed a white sculpture of an elephant or a wireframe of an elephant into one of these deep learning networks it is unlikely you will get the word 'elephant' as output. But if you feed it a real picture of an elephant it can correctly identify the elephant (assuming it was trained against photos of an elephant). Because the delta between a white sculpture of an elephant and an actual picture of an elephant is just color and detail this indicates to me that when you train these deep learning networks to recognize an elephant you are training the network to recognize details. It's a form of over fitting, the training is not general enough to catch geometry. It is correlating blobs of pixels , color and detail with an elephant rather then associating a three dimensional model of it to the word... the opposite of what humans do. In fact I bet you that if you took those very closeup photographs of an elephant and fed it into the network it'd do a better job at recognition versus the picture of a white sculpture of an elephant.
This indicates to me that to improve our vision algorithms, the algorithm must first associate pixels with geometry then identify the associated word to the geometry rather than try to associate blobs of pixels to words. Train geometry recognition before word association.
My guess is that our minds have specific and genetically determined built in geometry recognition algorithms honed to turn a 2d image into a 3d shape. We do not learn to translate 2d to 3d we are born with that ability hardwired. Where learning comes in is the translation of this shape to a word. Whereas most of the machine learning we focus on in research is image recognition, I believe the brain is actually learning shape and geometry recognition.
The problem described in the article is taken into account when building systems (like autonomous cars) using neural networks.
Where can I find more information on this?
Keywords are "control theory", "sensor fusion", "automated decision under uncertainty" that you can look on Google, Wikipedia and arXiv. Also "Simultaneous localization and mapping" which is a good example of using uncertain data points from different sensors to build a representation of the reality.
In those systems, a neural network is just another sensor providing augmented information.
Edit: I need to rephrase, the interesting case for me ( and the one the article is describing) is when the model fails with accurate input data due to assumptions that are intrinsic to the model itself.
As another comment said, "second take" is not what neural networks are made for. Neural networks are a building block of more complex decision systems, where the weakness of the neural networks are taken into account before automatically commiting to decisions.
Otherwise, I guess the article is good at pointing the current limitation of neural networks alone.
And while ML isn't all about NNs and deep learning, we're quite enticed by them right now and try to push them to see where their limitations are. So I think with a bit more generous interpretation the title is acceptable. I think it's certainly not clickbait, since NNs are right now a very significant part of the ML-research landscape.
This is science vulgarisation, or a magazine article. Not that there is no value in it, but it's an over-generalization.
I disagree, for the following reasons:
- The paper is called The Elephant in the Room. Technically the paper confronted the popular machine learning approaches, so the reversal would have been a more eye catching headline. "The Elephant in the Room confronts Machine Learning" is a rare opportunity to honestly play with word order in a way that would arguably be more attention-grabbing.
- Machine learning is more general than Convolutional Neural Networks, but less general than AI. This seems appropriate for this type of popular science publication.
- There are approaches that use attention and RNNs and they are an interesting approach, but by no means mainstream nor producing state-of-the-art classification accuracies. Actually, second take is exactly what an attention-based RNN can do.
- The last time I checked the state-of-the-art solutions to image classification involve Convolutional Neural Networks.
If there was an approach that was even close to approaching ConvNet performance, I would agree that it is perhaps unfair to condemn the entire field. Even then, we would be in the territory of generalizations in headlines, which are often fair. "{party} proposing legislation" is a probably bigger generalization of headline that would generally pass without criticism.
How is this argument related to the confusion of the terms Machine Learning and Neural Networks...?
> - Machine learning is more general than Convolutional Neural Networks, but less general than AI. This seems appropriate for this type of popular science publication.
The most appropriate term is the most precise one, in this case : neural networks
> - There are approaches that use attention and RNNs and they are an interesting approach, but by no means mainstream nor producing state-of-the-art classification accuracies. Actually, second take is exactly what an attention-based RNN can do.
Correct, but you are still talking about neural networks...
> - The last time I checked the state-of-the-art solutions to image classification involve Convolutional Neural Networks.
And again neural networks.
️
We're talking about the title.
> The most appropriate term is the most precise one, in this case : neural networks
That is generalizing RNNs and CNNs into "neural networks". If I take a popular CNN architecture, chop off the final dense layer and activation and attach an SVM classifier, is it an SVM or a convnet I am using?
Somehow the title "The Elephant in the Room confronts faster_rcnn_inception_resnet_v2_atrous_coco, faster_rcnn_nas_coco, ssd_mobilenet_v1_coco, mask_rcnn_inception_resnet_v2_atrous_coco, mask_rcnn_resnet101_atrous_coco." (or the reverse) doesn't have the same ring to it and is probably inappropriate for the audience.
There is nothing fatal here its just one more problem to solve. Havent we heard all about this issue of image recognition being trickable 1000 times, why is this the top post?