Aren't the backdoors gonna be in the baseband firmware that no phone allows you to modify?

I always felt that we basically "lost" with open source: Intel did the Management Engine and phones stuck everything in the baseband.

There's no real possibility of having fully software-hackable hardware. Someone else ultimately has control of all hardware, if they want to exercise that control.