>I do wonder though if we give the LLMs enough examples of texts with people describing their relative spatial position to each other and things will it eventually "learn" to work things these out a bit better

GPT-4's spatial position understanding is actually really good all things considered. By the end, 4 was able to construct an accurate maze just from feedback about the current position and possible next moves after each move by GPT-4.

https://ekzhu.medium.com/gpt-4s-maze-navigation-a-deep-dive-...

I think we just don't write much about moving through space and that is why reasoning about it is more limited.