Cell phones operate in licensed radio spectrum, so they need to have proper testing and certification (https://www.fcc.gov/oet/ea/rfdevice). Any device not properly certified would be illegal to manufacture or import into the US.
Separately cellular networks require PTCRB certification of the devices to ensure they are interoperable with the network (https://www.ptcrb.com/). The FSF could in theory write custom firmware for baseband and wifi chips, but they would need to seek certification as this would be considered a substantial modification. It would likely require cooperation from the chip manufactures to provide samples with various testing/debugging harnesses enabled.
Qualcomm and the like would probably sue to stop the FSF on the basis that it could put their own device certifications into jeopardy.
That isn't even touching on non-transmitting components like GPUs or sensors where the actual functional logic may be split between hardware and software (your blob driver). Even by doing a clean room reimplementation, you risk infringing on software patents, and will have little flexibility to work around them since the hardware will expect things to be done a specific way.
You would think it would be ridiculous to assume the people working on this know nothing about the mobile space, yet their actions do bring that into question.