Yep, focused on the play button on a YouTube video to get my mouse there and then my eyes immediately went back to the center of the video a fraction before I clicked the mouse button.
The problem is the Vision OS has no cursor. Cursor position is critical. Your eyes move around a lot more than your cursor moves, but in Vision OS, cursor position and eye position are the same. Very annoying. Maybe they could put a little cursor in the center of the screen that you could move around with another gesture.