Any idiot who breaks into your home would be well positioned to steal your server along with all of your memories and use them for ransom/blackmail/etc.
An encrypted hard disk with the key on a USB stick would be enough, just keep the key somewhere separate and you'll only have to plug it in when there's a power outage.