Hacker News Reader

23 TiB ... I would use 4 disks for that - each 10 TB or 12 TB in size (depending what room You want).

In RAID5(3) + SPARE with ZFS that would be 'raidz' mode.

    % math 12000000000000000 / 1024 / 1024 / 1024 / 1024
    10913.93

    % math 10000000000000000 / 1024 / 1024 / 1024 / 1024
    9094.94

From 10 TB disks You would have 3 X 9 TiB which means 27 TiB of space available.

From 12 TB disks You would have 3 X 10.5 TiB which means 31.5 TiB of space available.

> 1. I don't need the files to be accessible online, in fact, I would prefer if they were not.

I would keep it on a local LAN w/o Internet access.

> 2. If anything is backed up to the cloud, I want pre-internet-encryption with keys that only I know and control.

Use rclone(1) with its encryption - You can clone these files to S3 in the cloud.

> 3. I want something simple, that could be recovered using a pragmatic approach and open source software in case of a disaster.

I use rsync(1) for forever-incremental backups and rclone(1) to backup some of that into encrypted S3.

My rsync(1) scripts are here (maybe You will find them useful):

- https://github.com/vermaden/scripts/blob/master/rsync-delete...

- https://github.com/vermaden/scripts/blob/master/rsync.sh

> 4. I'd like a system where I can easily test my recovery strategy.

Also with rsync(1) ... or anything else for plain dirs/files.

> 1. What local filesystem setup should I use? Number of drives? Local backup approach?

ZFS.

Details about ZFS pool settings here:

- https://vermaden.wordpress.com/2023/04/10/silent-fanless-del...

> 2. If you've done this before, is there a strategy that you used for the actual aggregation of the data?

I am doing something similar for 5 TB data set. I have 4 data sets. I use 2.5 5 TB drives with ZFS and GELI encrypted.

    Main Source ==> Backup @ LAN (LOCAL)
             \ \
              \ > Backup @ Internet (SSH) (REMOTE)
               \
                \> Backup @ USB (OFFLINE)

> Are there any particularly convenient IDE to USB docks?

Many - check ALIEXPRESS.COM for tons of them.

> Any good software that you would recommend for locating duplicate files?

    % cargo install czkawka_gui czkawka_cli

You may also use ZFS deduplication for the datasets you KNOW have duplicated data. There is also new ZFS feature Block Cloning - You may want to look into that as well.

> 3. What remote backup software should I use?

I use rsync(1) and rclone(1). The rsync(1) for everything file/dir based. The rclone(1) to put encrypted backups into S3 containers.

Regards, vermaden

View on HN