What's missing from this discussion is how we humans don't just use a single, vision based "deep net" in order to identify the banana, we use lots of very different deep nets that run in parallel and cross-validate each other.
We have a vision net like computers that can be fooled with the same tricks (just google "optical illusion").
But a second net does three-dimensional inference of the banana shape from stereo vision and cross-validates that with a toaster. If in doubt, unnoticed parallax movement will be commanded to our body in order to get a better shape estimation.
Another net parses the content. A banana in a jungle or on a table would be highly plausible. A banana in an operating room or on a roller coaster would be implausible.
Another net parses context and timeline. If no banana was there and no banana has been incoming through our continuous estimated 3d space, there is probably no banana there.
There's smell, taste, all kinds of things running in parallel that continuously cross-validate our perception of this being a banana, probably a lot of different models just from vision alone that feed back into the main classification thread.
We'll even happily do looped cross validation if our eyes rest on a scene. Our memory will fetch reference data for all first order predictions and start subconciously processing this into the main prediction. It will predict how a banana feels, how soft it is, how warm. If we see a banana on lava that is not burning, our logic and reasoning will challenge the prediction. All of this happens subconciously, and there's no magic behind.
If you ask me, there are no limits of deep learning. There's just still a limit of imagination in feature extraction and processing.