How I wrote a script to generate faces by inverting Haar cascades with Python

Parts of machine learning used to be focused on density estimation / manifold learning for exactly this reason. It seems to be the case that supervised tasks (for sufficiently complicated and diverse problems e.g. ILSVRC, MSCOCO) implicitly learn some portions of what you would want from a density estimator. This is enough to generalize to new, but related tasks - see the huge body of work on pretrained VGG for {semantic segmentation, style, texture generation, depth/surface normal prediction}.

Learning a general density that is useful is quite difficult - you could be missing entire modes from the data distribution and never know it. You don't know what you don't know, after all. And there is still no clear (in my opinion) evidence that learning a density really helps generalize to a new task, especially when the end task is still a supervised one.

Explicit density modeling seems to help the most in semi-supervised / limited data settings, but any time you can bring some enormous but related labeled data to bear on your supervised problem it seems to win over any kind of density estimation tricks - compare trained from scratch nets for the tasks I mentioned, to ones exploiting pretrained VGG models.