There is a straightforward manner to overtake your phone number (call your carrier and use social engineering). There is nothing you, the customer, can do to lock that down. (I've tried with my carrier.)

With email, you can lock that down with robust 2FA (Google Authenticator/Authy/etc) and crooks have no straightforward way of defeating that.

This is how it plays out year after year and why SIM-swap gangs are so prevalent.