I know it is not a perfect solution but there is an Authenticator app that you can use on Android. I believe that iOS might have one too but I am not sure.

This solution does not require you to give them your phone number :).