It's probably worse than that. How would you guard against the threat of someone forcing you to unlock your phone at gunpoint? Call their bluff and just throw the phone in the gutter and run away? Stall for time like you forgot the password?
Yeah I don't think you can defend against a gunpoint threat, but my guess that's much less common than a casual petty thief who just snatches your phone out of your hand and runs.