Or put another way, a shorter sight radius by a factor of X makes it X times harder to visually confirm your iron sights are correctly aligned. This makes the visual-mental-physical feedback loop of keeping them aligned as precisely as you possibly can, much more difficult.

Shooting accurately is much more than just physically holding the barrel steady; it's a whole "symphony" of coordination, involving physical, visual and mental components.