Q: What's your password?
A: I have no idea.
Q: How do you log into computers then?
A: I'm going to invoke my right to remain silent now, thanks.
[I am under no illusion that this actually helps the compelled decryption situation]
Of course you cannot take back what escaped your mouth prior to asserting for right to remain silent.
It's also useful to be able to leave the physical key at home, or with another party.
... and if you hand it to your attorney, does that count as privileged communication?
I use my password manager to save things that are not passwords too- padlock combos, combo to my safe, medical info, parents wills, banking, all my dirty secrets... Seems wrong they could get access to all my info if the warrant is just for Xyz.