It is not just for the phone, the passphrase you set is also used to encrypt your data on the signal servers. Signal claims to use intel SGX to secure the passphrase on their servers, though.