The problem isn't that Google knows your phone number but that they will use it (or more precisely the attacker will use it after redirecting your texts) for account recovery which you can prevent by removing the number.