The goal of 2FA/MFA is to make you demonstrate that you're in possession of two independent secrets (authentication factors). Once you've shown that, it's considered safe enough to replace the second secret (OTP sent to your phone or generated by your TOTP app like Google Authenticator) with a cookie (the check is not IP-based). Typically the cookie only lasts for 30 or 60 days.
If what you're concerned about is the idea that Google knows your phone number, you can use Google Authenticator or another TOTP app instead.