Your friends will think you're crazy, but cats will love you.
It's precious.
Why fixate on the face and not, say, our hands or body, if they might be afraid of us, to determine if we are a threat? Eyes must provide a higher signal of intent, so it's surprising that even animals on a much lower level of intelligence can read our facial expressions.
I'm not sure if it's a good or a stupid question.
The face and eyes indicate intention. You can't successfully track or kill without focusing your full attention on the object, and the easiest way to detect this is by seeing where the eyes are focused.
Once a threat is determined, then focus moves to other body parts.
Think about it this way how can you kill successfully if you don't have a basic understanding of the anatomy of what you're trying to kill?
I think a general body map is hardwired into more vertebrates. Heads always have eyes, ears and move around. Arms are on the upper torso and legs are at the end of the torso, tails are common.
Beyond that level, your guess is as good as mine though.
If the individual means you harm, they're going to be paying close attention to your every movement since they're about to attack. They'll be staring directly at you, sizing you up, preparing to pounce.
Looking away, smelling around, yawning, slow blinking or eye closing, are all boondoggles that signal, as you said, "hey, I feel at ease with you enough to do this idle busywork, almost as though you weren't here at all because I don't see you as a threat or target".
That said, cats very much do stare at your face when they want or are expecting something, usually food. They also have the adoring eyes-half-open stare mode you can see examples of at r/Catsmiring.
I would say that cats better fit the introvert archetype, and dogs the extrovert one, rather than something from the DSM. Anecdotally there's also a significant positive correlation with their respective owners' personalities.
But not always. For instance, a common sign of flirting in humans is to repeatedly catch someone's eye and then look away.