Does nature tend to create forms which are easily detected by relatively simple neural networks? The evolutionary explanation could be that this allows animals with primitive neural systems to more easily distinguish other members of their species.
Ultimately, if you are able to look at an image and say "letter opener" there are features which differentiate this from a knife/can opener/whatever - these are the exact things a convolutional neural network should be able to use (in theory) and has nothing to do with the label, which is typically unimportant as long as it is unique and accurate.
We could flip all the labels around and still get unique answers - the network is just learning a mapping from input -> some integer, and I would argue the variance in dog breeds and lighting in natural scenes is much trickier than the angle/shape of a letter opener.
I still think it comes down to the composition of this particular dataset. Augmenting this with images scraped from online stores would be very interesting as it is fairly trivial to get huge numbers of images for anything that is typically sold online - I think Google is way ahead on this one!
It's a question of how visually coherent the category you're trying to learn is. From a purely visual perspective, the first two categories are relatively tightly bunched in the state space, whereas "dog" covers a diffuse cloud of appearances whose total range might even encompass the area where many non-dog animals also lie. Humans may rely on some additional semantic knowledge about different kinds of animal to produce an accurate classification. It's not entirely unlike how determining the meaning of the words in the phrase "eats shoots and leaves" can't be done reliably without incorporating contextual clues such as whether we were just talking about pandas or a murder in a restaurant.
There may also be issues around how distinct the categories are from each other. A couple years ago yours truly picked up a letter opener off the table and used it to spread butter on his toast, much to the amusement of his hosts.
In general, if you misclassify "dog" for a fixed architecture you will most certainly misclassify "Blenheim Spaniel" and "Flat-coated Retriever" - the two other classes are subsets of the first. The "eats shoots and leaves" sentence is analogous to a "zoomed in" picture of fur - we don't know what it is but we are pretty sure what it isn't! This is still useful, and would already get most of the way there for large numbers of fur colors/patterns.
I think the concerns you have are more important at training time, but I have not seen a scenario where it has mattered very much. In general having good inference about these nets is really hard, but I think your initial thought about "dog space" ties in nicely to a post by Christopher Olah (http://christopherolah.wordpress.com/2014/04/09/neural-netwo...) - maybe you will find it interesting?
And yes it becomes really fascinating to extend your last thought to "optical illusions" and other tricks of the mind - even our own processing has paths are easily deceived and sometimes flat out wrong... so it is no surprise when something far inferior and less powerful also has trouble :)
Isn't it contextual - flat-coated retriever, well-done, but how good is it at picking one out of a pile of images of black animals, panthers, house cats?
I know for sure that ImageNet has a huge amount of animals in it, down to very select sub-breeds, while the object categories are usually at higher levels.
Entropy isn't the right term here - low entropy would imply something about the compressibility of each image based only on whether it was a machine or a natural object, which I don't think is case.
I was using entropy in the information theoretic sense, as you might assign a measure of entropy in bits to each character in a language. If part of an animal is "less surprising" I'd say it contributed less to the entropy of the whole thing. Maybe that's too woolly.
There is a difference between complex mechanical and organic objects, though. See the universal machine tools illustrated on this page: http://www.lathes.co.uk/adcock%26shipleycombination/
There is no way you could identify the whole machine based on seeing a small part if an identical part recurs in many different machines. It's not holographic in the way animals are.
> Does nature tend to create forms which
> are easily detected by relatively simple
> neural networks?
It's probably the other way around. Neural systems optimize for natural objects.
(As you can tell, I know nearly nothing about neural networks, but curious to learn more...)
Deleted comment
There have been experiments trying to encode information the way the brain does, they just haven't worked very well (or at least not as well).
There is equivalency with PCA (well ZCA, a modified form of PCA) in cat and monkey brains, and likely others as well. See Sejnowski and Bell in [1]
Also, PCA is an affine transform, so there is no reason it couldn't be incorporated/learned by the net itself. In fact, I think most nets these days eschew PCA/ZCA when they have sufficient data support.
To clarify for others, this type of neural network has nothing to do with the brain. A neural network is really a "universal function approximator", and I actually prefer to call it as such. Our goal is to learn the best possible mapping of input -> label, through whatever means necessary. It turns out that learning hierarchies of features helps from both a learning aspect and a computational point of view. But a sufficiently wide single layer could do the same thing in theory.
It is. It's a family of models that (a) can be expressed compactly using linear algebra and (b) can represent a large class of mathematical functions. (In fact, the family of neural nets can learn all continuous functions.) "Learning" is typically some variety of gradient descent (on the "error surface" defined by the training set) in the space of parameters.
Are the computer-based neural networks so closely modeled after biological equivalents that they would inherit such an optimization?
In my opinion, no. Biological neural networks are, in fact, a lot more complicated. They have time behavior, changing network topologies, and chemical influences (neurotransmitters) that play a major role. There's still a lot we don't know about them.
I disagree with GP's contention. While the biological neural network was an inspiration for this class of mathematical models, and convolutional behavior (in ANNs, a regularization, or mechanism for reducing the dimensionality of the parameter space, sacrificing training-set performance but often improving test-set performance) may be used in our visual cortex, but artificial neural nets are quite different and, mathematically, most varieties are quite simple.