Say we have 10 images as a starting point. Then we create 10,000 images from those 10 images by adding noise, filters, flip them, skew them, distort them, etc. Isn't the underlying data the same (or some formal definition of shannon information entropy)? Would that actually improve neural networks?
I've always wondered. Is it possible to generate infinite data and get almost perfect neural network accuracy?
Basic answer is no, but the reason is kind of interesting.
Imagine that the input to the model is a list of facts, initially the facts are just:
* Image 1 has class A
* Image 2 has class B
* etc...
* Flipping the image does not change its class
* Translating the image does not change its class
* Adding a small amount of noise to the image does
not change its class
* etc...
So, by generating more data the model can progressively learn those "class invariant transformations", but the model would only reach perfect accuracy if all class invariant transformations were taught.
Another way to "teach" a model these rules is to embed the rule into the structure of the model itself, eg the idea behind convolutional neural networks is to embed translation independence into the model, so that it doesn't need to be taught that from large batches of translated images.
For example, most CNNs aren't invariant to skew, distortions, rotations, or even zoom level. So to train a neural net to recognize both 8x8 pixel birds and 10x10 pixel birds, you need to add images of both zoom levels.
Of course, this is a weakness, and there is a lot of research to try to rectify this, like Hinton's capsule networks.
For things like adding noise, in some cases it is used as regularization to make the model robust to noise, in others, such as GANs, models are trained to learn the difference between the generatively created images that are higher entropy from the true images, to refine the model.
But as the sibling noted, and you mention, the underlying data is somewhat the same, so yes, you do need a lot of diversity... but in practice these techniques can be helpful.
It's not a silver bullet. It won't capture the natural variations that happen in the real world.
But new forms of augmentation (like OP) are helping us get closer.
For example, MixMatch creates "mosaic" images by combining images across the training set [1]. In object detection, bounding box only augmentations are improving models by introducing variation [2].
And an anecdote: I work on https://roboflow.ai , and we've seen customers make production-ready results from datasets <20 images based on techniques like these.
[1] https://arxiv.org/abs/1905.02249 [2] https://arxiv.org/pdf/1906.11172.pdf
Data augmentation alone (e.g. rotations / shift / crops / color perturbations / cutout... of a single photo of an husky dog) will never yield the added information that is contained in new pictures showing subtle variations of the phenomenom your are trying to model (e.g. a new photo of a Dalmatian dog if you have no Dalmatian dogs in your original training set).
What you are talking about is data augmentation, a strategy we can use to expand our training dataset synthetically, mostly in a bid to prevent over-fitting.
The core problem in ML is generalization; simply put - how well does your approach work with new data it hasn't seen before. Think of it this way: there is a large set of all the potential inputs you could see, and you only get to see small subset when you are training; what do you do so your general performance is best? Which of course you can't actually know but you can try and estimate.
There are two issues that can give you a lot of trouble here. The first is overfitting (you'll do much better on the training set than "in real life"), the second is bias in in your training samples. Data augmentation (what you are talking about) is one approach to reduce parts of the former effect, and done correctly it can help.
Take a simple example, imagine we were trying to recognize simple geometric shapes on images of a page - you want to find triangles, rectangles, ellipses, etc. I only give you a small set images, say 10s of shapes total.
Now you suspect that "in the wild" you can have triangles at all sorts of rotations, and sizes, but I've only given you a few examples. So you want your algorithm to learn the shapes, but not the sizes or orientations. If you just train on these, it may not recognize a triangle that is just 2x as big as any it has seen, or rotated 20 degrees left from one it has seen, etc.
One approach would be to try and find a rotation and/or scale invariant representation for your inputs - if you "know" that shouldn't matter you've now removed it from the problem. This can be hard or even mathematically impossible to do, depending on the problem space (e.g. there is no rotation and scale invariant manifold for photographic images). So another way you can approach it is empirically; to take the examples I gave you, and generate new examples in different poses and scales. You feed this into your training and should get a much more robust result, one that doesn't hew too closely to the training set (i.e. less over training).
So this sounds great, right? What could go wrong? There are a few issues. One is you are now enforcing things outside what you learn from the data, so if you are wrong you will make things worse.
More subtly when you do this you can tend to amplify any of the sampling biases you had originally. Imagine that I never gave you an equilateral triangle in the training set. It's quite plausible that by generating millions of inputs from a few examples, this category gets pushed closer to something symmetric, like a sphere , say.
Another issue that can be subtle is that the manipulations you are doing for data augmentation can easily introduce new things to the data that you don't see, and your training can pick that up. Consider, for example, rotations of these shapes. I told you we were doing this from images, i.e. discretely sampled grids. This means that other than certain symmetric rotations and flips, you can't do this without resampling. And you can't resample without smoothing. So if you take a dozen or so "crisp" examples and turn them into 10s of thousands of "smoothed" examples, what exactly are you teaching your model. I'm also waving my hands hear about how you are extracting "shapes" from "background" and in a NN context, what your inputs actually look like... but you can introduce issues here also.
There are lots of trade offs here. It's a useful technique, but unsurprisingly isn't a silver bullet.