One thing to note is that here noise != Gaussian iid noise, so these are not typical white noise images. I think we were not really clear on that part, but for us noise is basically a random process, which takes a seed as input (plus potentially some very low-level assumptions over image statistics, such as a 1/f spectrum) and produces a synthetic image.

It is then possible to generate arbitrary amounts of these images as samples from the stochastic process - these images exhibit certain image-like structures (such as oriented edges), but are as a whole still random and extremely varied, which is good and necessary for the representation learning.

In terms of helping, though, it is important to note that we do not achieve state-of-the-art performance yet, and when looking at absolute performance for a task like image classification, using real images is still better. That being said, something that is in the paper but generally seems to get lost is that our representations work very well when analyzing data that is very different from normal images, such as medical images or satellite images.