Unsupervised techniques work really well for language modelling.
There is also weakly supervised and distant-supervision, where the labels are "noisy" or not exactly what you want.
You're right in that strong supervision, where you basically trust your class label, works really well, because it's probably the easiest case.
Combining unsupervised (e.g. pre-trained language models) with a very small set of strongly labeled data, or a larger set of weakly labeled data, seems to work pretty well too.
(Because if you know a priori what is it that you want to measure - it's supervised)
One of the features was a subjective rating of how much I liked some of the women, and scikit-learn then suggested to me other women in the clusters that had my best ratings. It turns out that I like vegetarians, redheads, and left-wingers. Which happens to be true, even though I eat meat and do not identify as left-wing. But those traits correlate with _other_ traits that are more difficult to measure objectively, such as caring about children, liking to hike, and preferring an evening of sex to an evening of television.