The attacker could be pressing the button, or even opening your eyes for you :)
And if someone has full physical control over you such that they can open your eyes without consent, do you really care if they can unlock your phone? Your life is in their hands at that point anyway.
You have to assume a persistent attacker with physical access will be able to crack the device regardless.
It only works if you're close, alive, your eyes are both open, and looking right at it. I doubt that degree of specific physical attack is in most people's threat model. It's only backing a 4/6 digit pin for most people anyway. Realistically, it's not the weakest link.