Self Supervised Learning in NLP

The deepemoji one is fooled by "was my flight delayed? no.". I feel for the computer when it meets that one "do I speak in questions?" person. chuckles.

On a more serious note, Hinton and other alluded to the need to restructure NLP studies as to focus more on the nature of recursion within language, which is basically what Chomsky has been saying for decades. It's interesting to see whether they converge.

It’s like one of those low effort medium articles. But not low effort, and not on medium. I really enjoyed it.

Thank you - I found the article helpful in giving me a lay of the land

Hello, number 9) doesn't say what the task is?

Also, I always wondered, do those methods work universally on all languages? For example Chinese, Korean and Japanese, with different alphabets.

I'm wondering if these tasks have a form of bias that decreases the performance. If the model sees only positive examples and no negatives then it is biased on the positive paths of decisions. The moment where one changes the path to be incorrect, the model can't recover from the mistake because there weren't any negative examples during pretraining. There's many words that never follow some words but the model never sees that.