On iOS at least, it gains affirmative consent by you double clicking a button on the side. It also refuses to recognise your face if your eyes are closed.
And if someone has full physical control over you such that they can open your eyes without consent, do you really care if they can unlock your phone? Your life is in their hands at that point anyway.
You have to assume a persistent attacker with physical access will be able to crack the device regardless.
It only works if you're close, alive, your eyes are both open, and looking right at it. I doubt that degree of specific physical attack is in most people's threat model. It's only backing a 4/6 digit pin for most people anyway. Realistically, it's not the weakest link.