In particular, what's it like using the phone outside of the silent circle ecosystem...because let's face it, 95% of the people you are going to call/text etc will not be using a similar device
I would assume it would have the same effect as with end-to-end encryption mails, where you just have unencrypted data when you are communicating with people that don't support this. Such as is the case with https://protonmail.ch/
your side is encrypted to the telecom systems switch hosted by silentcircle, unencrypted outbound from there.