The semantics they are trained on include latent representations of the world that are superior to any pre-existing PL corpus. That is my guess and it's not any less rigorous than your guess.