With the current availability and prices of Raspberry Pis, you might be better off getting a PinePhone as a Linux phone.
https://skysedge.com/unsmartphones/ might have a more interesting BoM to look at if one wants to do things from scratch, since a bulky pi-specific board might be too much for most