Generally, not legally without a warrant, which can be a good enough deterrent. Or at least should be if our legal process is working at all.
It's more like burying treasure. The location is a secret map (key) that only you know. The critical distinction here, I believe, is between "having" and "knowing".
And I like to believe we have even stronger rights to what we know than to what we have.
All that said, this is assuming use of working cryptography. With the exception of the technically savvy (those who know how to hid things properly) asking for backdoors into encryption is akin to registering your treasure with the feds, something I'd assume gold miners wouldn't have put up with, for example.