They can really easily do that though. A court can say "give us your keys or you are going to sit in jail until you do and we will record your conversations so if you tell the keys to someone else, we'll sell them"
So? If sums is high enough they STILL can't confiscate it. In some cases it is preferable to spend some time in jail versus giving up outrageous fortune. They still can't take it by force. Also, only a few countries with backward laws (like the United States of America) have laws that allow that. Something insane like what you propose would never be possible in a developed country with a solid judicial branch like Estonia, for example.
It's not much in its favor, but a sufficiently motivated person can resist handing over cryptocurrency better than they can any other asset.
If I've got $20 bill or a gold bar in my pocket, I can use it at the store if I can get to the store, but someone could easily confiscate it from me. If I have a $20 bill or gold bar buried in the woods behind my house, it can't be easily confiscated from me, but I can't use that at the store, even aside from whether I can access a store. If I have a cryptocurrency wallet, I can use it in the hypothetical cryptocurrency store if I can access it, and it also cannot be easily confiscated from me (assuming reasonable security measures).
The real criticism is "what cryptocurrency store?" and that's the real reason cryptocurrency isn't very useful. I'm not arguing that cryptocurrency is good or useful, I'm arguing very narrowly for one specific point: "owning cryptocurrency" is more like knowing something than it is like possessing something.