I think most Americans know that the government can quite easily enter anyone’s home if they want to.
It's more like burying treasure. The location is a secret map (key) that only you know. The critical distinction here, I believe, is between "having" and "knowing".
And I like to believe we have even stronger rights to what we know than to what we have.
All that said, this is assuming use of working cryptography. With the exception of the technically savvy (those who know how to hid things properly) asking for backdoors into encryption is akin to registering your treasure with the feds, something I'd assume gold miners wouldn't have put up with, for example.
There are two main issues: government abusing its power, and weakening security of everybody. I meant to focus more on the second.