1. Phone number requires a service provider
2. Service provider owns the baseband and root of your phone.
3. $government makes a "lawful request" of Verizon et. al.
to install keyloggers/screenshotters on your phone.
4. Your "secure" commms are compromised.
Signal needs to work on comprehensively rooted devices connected only via wifi for the above threat model to go away.