I also tend to stare at the mouth of the person I’m talking to!
Which, I think, makes it look like I’m looking at their chest because the women I talk to almost invariably adjust their shirt in a way that suggests they’ve become uncomfortable (because they think I’m looking at their chest)