This high dimensional vector space does allows them to theoretically embed a interpretable "deeper understanding", reference https://www.youtube.com/watch?v=wjZofJX0v4M&t=624s. It's the pre-train process that determines whether or not they demonstrate this "deeper understanding".

This "deeper understanding" refers to the understanding to the word, can be contextual, pragmatic, or semantic.