You're right, but I posed this very question to Kevin Mitnick once. He had just recounted how, during his prosecution, he'd been forced to give up his passphrase. (This was several years ago, and so I might have some specific details wrong, and I'm also going to be a bit vague about it as a result.) I asked him why he didn't just plead the 5th. He said he tried, but that they moved him into solitary and they started abusing him in jail. (abuse is my word, I don't remember the specifics) and they they used every method they could to force him to give up the passphrase, including, I believe, holding him in contempt for not revealing it. Eventually he revealed it.
It is my opinion, having read accounts of people being prosecuted by the government for various crimes over the past 20 years, that in many cases, the government does not hold itself within the bounds of the law, and that such prosecutions are often illegal themselves. Yet, they are able to get away with this, because, what prosecutor is going to destroy their career by pressing charges on cops, other prosecutors or judges?
Also, in this day and age, asset forfeiture is extremely easy. The government can seize all of your assets, effectively denying you the ability to pay your legal counsel, forcing you to rely on the legal services they provide for you. Further, even if you can afford your own counsel, as an "officer of the court" they are subject to extreme pressure. I've read about a fair number of situations where lawyers would not even introduce constitutional arguments for their defendants innocence because it would result in recriminations against the lawyers by the judge. (e.g.: its "frivolous" to claim that federal drug laws are unconstitutional, even though to criminalize alcohol an amendment was needed.)
I'm not a lawyer, of course, this is just my perspective.