source: For my masters degree I wrote a paper on the grounding problem (linking words said to physical objects), specifically the spatial reasoning component ("please give me the book on the table in the corner). We were able to get very simple commands to work, but anything remotely complicated, especially with a temporal aspect or things that can't immediately be seen (keep going until you see the red building, then turn left) is the realm of future research
Relative size, position, etc can be tricky to determine as they aren't just classification problems but require segmentation and spatial reasoning. Especially taking into account any difference in perspective between the user and the robot, as well as context (what does "above" mean on a map vs in a tower of blocks, or what does "left" mean if you are facing me)
Starship Robotics has a pilot in Milton Keynes in the UK for their delivery robots, and came across the problem of pedestrian crossings where you need to push a button to change the traffic lights & cross. Rather than engineering a solution that required changing the crossings or putting some complex robot-arm on to push the button, they just play a pre-recorded message begging humans to press the button for it.
It's easy to devise complex solutions which you can't implement - but it sometimes involves a bit more creativity when you constrain yourself to simple and implementable solutions!
https://arxiv.org/abs/2207.04429 https://arxiv.org/abs/2204.01691#google https://arxiv.org/abs/2207.05608#google https://arxiv.org/abs/2207.05608#google https://arxiv.org/abs/2109.01115 etc
This is a fairly accurate description of a "guide" i got from a nice guy after asking him where my hotel may be... and now imagine the software that must be involved to decode THIS...