Isn't it a general rule of driving (or operating any sufficiently dangerous machinery) to keep the eyes on the road, constantly reminding oneself to do so, so the attention is kept where it is needed? I mean, in theory. In practice, I see people deep in their damn phones all the time - and it's scary - but I think that's more of an attitude (social) issue than a display (technology) problem.

And, yes, surely, one needs to periodically switch attention to mirrors and instruments, and I must imagine that shorter gaze movement distance shouldn't hurt. It's the same as checking the speedometer - you don't see the road, only have a rough idea from the peripheral vision.

Although I can imagine that a HUD can be actively distracting, constantly intercepting attention, e.g., flickering.