I'm not going to mention the application software or OS, since I'm sure someone more knowledgeable than me will give you information about that.

What you're probably not thinking about is the baseband processor, which negotiates cellular connections and handles everything related to making the phone an actual phone. The trouble with baseband modems/processors is that none of them are open source and that they oftentimes have direct memory access to the application processor. iPhones and the Purism Librem phone isolate the baseband further from the application processor than the average Android phone.

Even if you find a nice phone and install an open-source, audited, secure operating system, your hardware might compromise you.