Same logic says if you have a safe full of papers --- records of things you know --- you can refuse a search warrant for the contents of your safe.
(Yes, that's my point.)
(Yes, that's my point.)
I think there is actually precedent (in the US) here, but I can't find the reference. I will keep looking.
Edit: see fatdog's comment for why.
The 5th amendment right to avoid self-incrimination only extends to testimony. A search warrant can compel you to furnish the contents of a safe (as property, it is governed by the 4th amendment -- due process is all that is required, and in this case, the process of getting a search warrant is all that the Constitution requires).