I dont agree with this, either we know what the pretext task was, or we fine tune with labels, but in either case, it's no different to analyze than supervised models.

I do agree that more parameters and a bigger training set makes the models more opaque (or at least more expensive to understand), but the self supervision is not the reason.