I can use my phone number for 2FA and/or as a recovery phone number. Would you advise to remove it from both places or just from 2FA?
A stolen phone can of course be a problem as well, but at least it's somewhat under my control and I may notice pretty quickly when it's gone.