Linux phones exist but work on a limited number of devices (see PostMarketOS) or Linux primary devices like Purism or Pinephone (which I've heard are expensive). That's the primary issue, you either need to buy an expensive and potentially underpowered phone OR have one that is supported most of the way with PostMarketOS (Wifi, SIM card, GPU, etc. sometimes may not work even though the phone can boot PostMarketOS).

I personally don't have a problem with a mediocre-performance phone but it should not be expensive. Not to mention app ecosystems - there aren't a lot of Linux apps for Linux phones.