I'm not defending this highly disturbing practice, but the point is that it's a political test. If they find anything politically disturbing on the phone, they can refuse you entry.
No they can't. Refusing to allow a US citizen to (ever) re-enter the country isn't allowed. At worst they can detain you.
However, CBP does not have the right to demand that you give up your passwords as a citizen. They can attempt to coerce you through detention (but if that detainment were prolonged you'd have a solid habeas corpus case).
[0] https://www.eff.org/wp/defending-privacy-us-border-guide-tra...