Where we see shapes, AI sees textures

But the insight is that although both features are informative, classical training approaches result in networks that over-fit on texture information and ignore shape information.

Table 2 of the discussed paper finds that joint training with "Stylized ImageNet" (IN images after a style transfer) and classic IN followed by a fine-tuning pass on IN results in improved classification accuracy over IN training alone.

When it comes to "shape", part of the problem is definitional. In the MNIST example, the network will fairly easily learn features that are nodes, like where two lines intersect, but it has trouble with paths, like "an unbroken line that smoothly intersects itself." CNNs have trouble distinguishing things like a spiral from a set of concentric circles because locally they look the same.

TL;DR; Logical reasoning isn't based on a "prepondernace of informative." Human-like image-recognition seems, to me, to quire both sort-of statistical reasoning, which depends on the preponderance of information and logical reasoning, which doesn't (not directly).

That's seems sort-of true but "Informative" has a number of different implication. It seems pretty inevitable that a huge image dataset is going to carry the most data in the form of textures.

The thing, a human being can (often) extract considerable meaning from an image of a stick figure or a black and white outline of a big cat, neither of which involves that much data. Effectively, human image recognition is able to both use texture-qualities as a rough guestimate for the source of an image but also look at the overall image structure and see "differences that make a difference". I mean, I can look at a puzzle piece and determine it probably goes in particular area but that's only one facility, "knowing" what an image "is", involves more.

I think MNIST is too simple to have much bearing on the problem. MNIST shapes are two-dimensional; actual object classification requires 3D shapes (and all possible projections thereof), which is a much harder problem.

Since this is just a message board discussion, my wild conjecture would be that the model has no idea about the existence of the third dimension, or basic physical concepts like lighting and distance. And without that larger world model in which to situate things, texture is maybe the easiest thing to cling to.

Actually I don't think humans vision is based only on shapes or textures. There are many novel shapes of planes and we know that's a plane no matter what. We use understanding, with our knowledge of the world, not only the information coming from the eyes. I think it is much larger than what is encoded in a deep learning neural network.

For example https://www.researchgate.net/figure/Visualization-of-example..., where you can see (somewhat, if you zoom in) that layer 1 neurons are interested in very simple features, like strong horizontal edges, or particular gradients.

Lighting changes indicating edges

Gradients indicating a smooth curve

Texture maps that discern between matte, smooth, translucent, often in depth enough to determine the texture material itself.

Reflectivity

Changes in reflectivity when reorienting oneself or the object

Binocular Vision

Parallax effect between frames

A large database matching objects to memories and experiences

Many other techniques for tracking and seeing movement.

In the above, Parallax and Binocular vision, and consulting one's memory are the only ways to determine the sizes of the objects being looked at.

It's why those optical illusions tiny/big rooms work so easily; neither we nor CV algorithms would be able to discern the sizes of objects without prior memory of such objects or by walking around it.