I totally agree with you. That was a mistake on my part . I had my number on the account a few months back.But my thinking was similar to what @orblivion posted below

"Because I've been warned by security conscious people never to use phone number as a 2FA because it's so insecure, and using it as a recovery option seems to create an even weaker link.". I happened to read so many articles in the recent past of SIM Swapping, that I was afraid to use my cellphone number as 2FA and removed it from my gmail account. What with Google being evil and all