For those of us out of the loop, could you summarize the idea of contrastive learning as a whole?
And Lilian Weng blog on self-supervision [2]
.. CPC is .. translating a generative modeling problem to a classification problem... uses cross-entropy loss to measure how well the model can classify the “future” representation amongst a set of unrelated “negative” samples...
[1] https://ankeshanand.com/blog/2020/01/26/contrative-self-supe...
[2] https://lilianweng.github.io/lil-log/2019/11/10/self-supervi...
To succeed, the encoder needs to be able to extract the underlying, useful information (called slow features) contained in the patch and discard the noise as this will make the retrieval process much easier.
This yields an encoder that gives pretty good representations of your inputs and you can then finetune some additional layers on top of it for your final task.