That person grabbing your phone would also need to know your password, so it's not any worse than not having two factor authentication.