The problem is not with the Linux phones that are out there, it is that no apps are and likely never will be compatible with (non-Android) Linux phones. Until that is solved, most people (like me) could not switch out their phone for a Linux phone as a daily driver, however much they would like.
Having Linux as an alternative for Windows is a lot easier than having Linux as an alternative for Android/iOS. A lot of the Windows applications we use can be replaced by opensource applications on Linux. And browser based applications already work on both.
However, apps on Android/iOS are mostly tied to cloud services. Unfortunately services that most people need in this age. Messaging systems like WhatsApp and Signal are usually tied to your phone, and going without them would result in social isolation for a lot of people. But also for banking and sometimes government services, you likely need to use their apps. And there will never be opensource alternatives for such apps.