You’re assuming the tracking is all visual but it’s not. Uses vision combined with accelerometer, gyro, etc. Which are affected by vehicle movements.

Travel mode apparently switches it to track primarily with vision. So it can work as you say.