I'm confused. Are they just saying if you throw noise-like images at your hot dog classier and label them 'not hot dog' it improves performance?
For example, to classify a hotdog it might be useful to first generate an intermediate representation of the image (think "cylindrical, brown, meaty thing"). Such a representation can then fairly easily be mapped to the concept "hot dog".
These representations can be learned from large image datasets alone (they do not require labels!). In our work we show that you don't even need real images, but that images that are generated from noise processes are enough to train such representations, and that these representations are surprisingly good for classification.
Hope this clarifies things a bit, and happy to answer any other questions!
Back before deep learning, people used to make recognizers for features like that as a lower level of feature recognition. Now it's expected that features will be derived automatically from real imagery. This is kind of a return to that level.
A useful training set might be a big texture library used for game development or animation. Those are easily available.
I would expect it to be somewhere in the ballpark of our StyleGAN images, which also look very "textural", but lack these effects that are an result of imaging the 3D world. Interestingly, modelling these effects without realistic textures seems to result in worse performance - this is for example the case for images taken from CLEVR or generated from Minecraft, and both perform worse than the StyleGAN images.
It is then possible to generate arbitrary amounts of these images as samples from the stochastic process - these images exhibit certain image-like structures (such as oriented edges), but are as a whole still random and extremely varied, which is good and necessary for the representation learning.
In terms of helping, though, it is important to note that we do not achieve state-of-the-art performance yet, and when looking at absolute performance for a task like image classification, using real images is still better. That being said, something that is in the paper but generally seems to get lost is that our representations work very well when analyzing data that is very different from normal images, such as medical images or satellite images.
But collecting lots of image data is hard so someone comes along and points out that the most important characteristic of your images is that they're appear drawn from an autoregressive process... That is that each pixel is (say) 95% correlated to the pixel 1 away, 95%^2 to the pixel two away, 95^3 three away and so on. So to compute the PCA you only need the covariance matrix anyways, so you just generate an AR0.95 covariance matrix and use the transform derived from that. And you find this works pretty well too.
This work is along those likes, they're generating random images with some simple natural-like statistical properties and training the first part of the classifier using them and getting useful results.
One interesting promising part is that this line of thinking may result in better network designs or insights that allow skipping training these initial layers: Going back to the block transform example, the next step would be to notice that the PCA of the AR0.95 matrix is the discrete cosign transformation, which is agreeable to an extremely efficient implementation.
The abstract seems to say that, but again, I could be misinterpreting: “Our findings show that it is important for the noise to capture certain structural properties of real data but that good performance can be achieved even with processes that are far from realistic.”
edited: Typo,clarity