I think you are wrong. We all have things to hide.
A joke between friends taken out of context. Your angry messages to a spouse or parent after a fight happened between you and them could be used out of context also. And what about your YouTube history and playlist or your web history...
> just your own home
Your home can be searched and all your devices at home can be seized as well. Granted it's probably a higher threshold than having everything on your phone but it's no a panacea either.