Low-Dimensional Structure in the Space of Language Representations

Huh. Based on a quick skim, these guys claim that:

* There is low-dimensional structure within the space of representations learned by language models.

* This low-dimensional structure is reflected in brain responses predicted by model embeddings -- specifically, they say they can recover known language processing hierarchies from the principal dimension of representations, and that the embeddings can be used to predict which representations map well to each area in the brain.

I'm going to read the paper carefully.