Illustrated FixMatch for semi-supervised learning

https://amitness.com/2020/03/illustrated-pirl/

But you are correct! This way of showing off your work is much nicer than what ends up in the paper. The blog representation is a good opportunity to show off the results of the paper on a higher level. However, the "mathy" paper is still important so other experts in the field can understand the details of the technique

Wanted to clarify that this is the summary article of the paper. I wrote it to help out people who might not have the math rigor and research background to understand research papers but would benefit from an intuitive explanation.

The actual paper is available here: https://arxiv.org/abs/2001.07685

Basic idea is to learn generic image representations without manual labeling and then finetune that on your small dataset. These are relevant articles I have wrote on it: https://amitness.com/2020/02/illustrated-self-supervised-lea...

https://amitness.com/2020/03/illustrated-simclr/

They augmented text by using backtranslation. Basic idea is you take text in English, translate that to some other language say French and then translate back the French text to English. Usually, you get back an English sentence that is different that the original English sentence but has the same meaning. Another approach they use to augment is to randomly replace stopword/low tf-idf(intuitively say very frequent words like a, an, the) with random words.

You will find implementation of UDA on GitHub and try that out.

I am learning these existing image semi-supervised technique right now and the plan is to do research on how we can transfer those ideas to text data. Let's see how it goes.

If you cannot do that due to them not including a vocabulary file, you're stuck using something like word2vec/FastText which is fine but not ideal if you're looking for grammatical correctness...

What you are talking about is data augmentation, a strategy we can use to expand our training dataset synthetically, mostly in a bid to prevent over-fitting.

Basic answer is no, but the reason is kind of interesting.

Imagine that the input to the model is a list of facts, initially the facts are just:

   * Image 1 has class A
   * Image 2 has class B
   * etc...

   * Flipping the image does not change its class
   * Translating the image does not change its class
   * Adding a small amount of noise to the image does
     not change its class
   * etc...

So, by generating more data the model can progressively learn those "class invariant transformations", but the model would only reach perfect accuracy if all class invariant transformations were taught.

Another way to "teach" a model these rules is to embed the rule into the structure of the model itself, eg the idea behind convolutional neural networks is to embed translation independence into the model, so that it doesn't need to be taught that from large batches of translated images.

Data augmentation alone (e.g. rotations / shift / crops / color perturbations / cutout... of a single photo of an husky dog) will never yield the added information that is contained in new pictures showing subtle variations of the phenomenom your are trying to model (e.g. a new photo of a Dalmatian dog if you have no Dalmatian dogs in your original training set).

For example, most CNNs aren't invariant to skew, distortions, rotations, or even zoom level. So to train a neural net to recognize both 8x8 pixel birds and 10x10 pixel birds, you need to add images of both zoom levels.

Of course, this is a weakness, and there is a lot of research to try to rectify this, like Hinton's capsule networks.

For things like adding noise, in some cases it is used as regularization to make the model robust to noise, in others, such as GANs, models are trained to learn the difference between the generatively created images that are higher entropy from the true images, to refine the model.

But as the sibling noted, and you mention, the underlying data is somewhat the same, so yes, you do need a lot of diversity... but in practice these techniques can be helpful.

It's not a silver bullet. It won't capture the natural variations that happen in the real world.

But new forms of augmentation (like OP) are helping us get closer.

For example, MixMatch creates "mosaic" images by combining images across the training set [1]. In object detection, bounding box only augmentations are improving models by introducing variation [2].

And an anecdote: I work on https://roboflow.ai , and we've seen customers make production-ready results from datasets <20 images based on techniques like these.

[1] https://arxiv.org/abs/1905.02249 [2] https://arxiv.org/pdf/1906.11172.pdf

The core problem in ML is generalization; simply put - how well does your approach work with new data it hasn't seen before. Think of it this way: there is a large set of all the potential inputs you could see, and you only get to see small subset when you are training; what do you do so your general performance is best? Which of course you can't actually know but you can try and estimate.

There are two issues that can give you a lot of trouble here. The first is overfitting (you'll do much better on the training set than "in real life"), the second is bias in in your training samples. Data augmentation (what you are talking about) is one approach to reduce parts of the former effect, and done correctly it can help.

Take a simple example, imagine we were trying to recognize simple geometric shapes on images of a page - you want to find triangles, rectangles, ellipses, etc. I only give you a small set images, say 10s of shapes total.

Now you suspect that "in the wild" you can have triangles at all sorts of rotations, and sizes, but I've only given you a few examples. So you want your algorithm to learn the shapes, but not the sizes or orientations. If you just train on these, it may not recognize a triangle that is just 2x as big as any it has seen, or rotated 20 degrees left from one it has seen, etc.

One approach would be to try and find a rotation and/or scale invariant representation for your inputs - if you "know" that shouldn't matter you've now removed it from the problem. This can be hard or even mathematically impossible to do, depending on the problem space (e.g. there is no rotation and scale invariant manifold for photographic images). So another way you can approach it is empirically; to take the examples I gave you, and generate new examples in different poses and scales. You feed this into your training and should get a much more robust result, one that doesn't hew too closely to the training set (i.e. less over training).

So this sounds great, right? What could go wrong? There are a few issues. One is you are now enforcing things outside what you learn from the data, so if you are wrong you will make things worse.

More subtly when you do this you can tend to amplify any of the sampling biases you had originally. Imagine that I never gave you an equilateral triangle in the training set. It's quite plausible that by generating millions of inputs from a few examples, this category gets pushed closer to something symmetric, like a sphere , say.

Another issue that can be subtle is that the manipulations you are doing for data augmentation can easily introduce new things to the data that you don't see, and your training can pick that up. Consider, for example, rotations of these shapes. I told you we were doing this from images, i.e. discretely sampled grids. This means that other than certain symmetric rotations and flips, you can't do this without resampling. And you can't resample without smoothing. So if you take a dozen or so "crisp" examples and turn them into 10s of thousands of "smoothed" examples, what exactly are you teaching your model. I'm also waving my hands hear about how you are extracting "shapes" from "background" and in a NN context, what your inputs actually look like... but you can introduce issues here also.

There are lots of trade offs here. It's a useful technique, but unsurprisingly isn't a silver bullet.

(Submitted title was "Semi-Supervised Learning: 85% accuracy on CIFAR-10 with only 10 labeled images")