Are you not aware of cold storage? Do you think people store millions on their laptops? Smart people use traditional means to store their crypto, literally paper in banks, custodians with insurance, actual vaults, etc, etc.
So if someone is able to guess your key, or just happen to find it, or if the encryption has been defeated by a leap in computing or algorithms, they will have access to your crypto without needing to access your key.
Thus someone accessing some of your keys would not help them at all (and other things can be helpful lile time locks in combination with other features)