Building NAS with ZFS, AFP/Samba for Time Machine (2021)
blog.gwlab.page
blog.gwlab.page
It used to be that after a while (varies from a couple of weeks to months) Time Machine would complain that the spare disk image on the NAS was corrupted and you'd have to start from scratch, which completely defeats the purpose of backup.
Were there any specific tweaks you made to your setup to make this reliable?
I also wonder if the amount of backup data makes a difference. I have tended to use Time Machine to backup pretty much my whole drive. I wonder if it's more reliable when backing up smaller amounts of data perhaps...
As to amount of data, I run it on my whole drive with a few excluded directories, that's 700-ish gb of non-excluded data and the backup on the Synology is 1.6TB. I've recently migrated the TimeMachine backup from an older Intel MBP to a new M2 Air (I wasn't even aware you could do that) and that seems to have worked just fine as well.
I would assume that'd work fine, because I run my main workstations and a server full time with my home folder and various data living off APFS or HFS formatted iSCSI targets served over 10 gigabit from a TrueNAS system. I haven't upgraded to macOS 13 yet since I tend to be a lot more conservative these days about Mac upgrades than I used to, but Xtend SAN at least still works beautifully under macOS 12. I really, REALLY hope Apple doesn't remove that capability without a native alternative (or at least officially sanctioned way to do the same thing). Of course in that specific scenario Time Machine is of limited value (and I don't use it) since why bother when you can just use ZFS snapshots on the NAS side instead. TM has always been too flaky for me to feel comfortable with anyway.
Still, could be powerful in some instances, particularly when combined with a hardware gadget setup. Which leads to a response to sibling thread:
>Asmod4n 3 hours ago: Can’t do full system restores from iSCSI
Linux has kernel support for USB gadgets that can emulate mass storage devices, and that means you can actually rig up a little cheap embedded board to translate iSCSI into something that will look to anything else like a standard external USB hard drive. See for example [0]. Like the author there I use it to work around issues with appliances like consoles or the like that don't know how to natively deal with a lot of network stuff, but it's a pretty powerful tool if you've got a good NAS and LAN setup regardless. Get your full network storage flexibility, snapshots, data integrity and so on that pretty much anything ever with a USB port can utilize natively.
----
The pool itself is 8x12TB Toshiba enterprise drives. I'll have to check what pool config I used but I did add multiple NVME SSDs for SLOG and ARC. Whichever one is the important one is actually mirrored across the NVME drives for redundancy while the other partition only exists on one NVME drive, which was mildly annoying to configure.
I'm still not sure I made the "best" choices for the ZFS config - there are many knobs to turn.
Connections are a mix of wired and wireless. The NAS box has 10GbE to my UDMPro.
See my other comment for how the datasets are setup. I forgot to mention that I applied a quota to force each client to prune its history to avoid consuming all disk space.
In terms of stability, I haven't had a problem with TM on a NAS for a few years, but yeah the corruption is why I used to use CCC instead.
I'm using ZFS + netatalk in Gentoo server since 2015 as a backup server for 5 macOS devices without problems.
Early on I had an issue with over provisioning the ZFS slice or whatever but that was resolved.
So far I've found it to be pretty reliable.
I've given up on Time Machine over SMB and I'm using Borg + Vorta for my backups. It has the benefit of being readable by any other Linux machine, too.
I'm not sure what causes TM to start considering the volume corrupted, because I believe at one point I switched over to a hardwired connection, but still saw a corrupted volume.
I compared that to Borg, which doesn't flinch if you SIGKILL the process mid-backup, and will basically just pick up from where it left off if you start another one. I saw no reason to continue with TM at all.
FWIW, I've done Time Machine backups over Wi-Fi to a QNAP NAS for several years and have never run into this issue. I'm just using default settings for everything.
Thankfully this is not my first rodeo. My Time Machine over SMB on ZFS setup has zfs-auto-snapshot set up, so I was able to roll back to a snapshot a month or so prior, which Time Machine happily verified, and it resumed backups without a hitch (keeping several years worth of backups intact).
I also found increasing the band size to 50MB from the default of 8MB made things work “better”, but I lack concrete empirical evidence.
If the hardware is remotely capable I’d probably advise running proxmox and spinning up a turnkey Linux fileserver container with bind mounts to the underlying zfs filesystem to use as the main samba share manager. This makes it easier for less technical people to manage the users and permissions and stuff with a web gui. I had some trouble getting zfs sambasharing to work with sharesmb=on (not sure where those configs are stored on proxmox, and if it can do everything samba normally can), but you can try that route to or use sharenfs to set up NFS shares.
There are a bunch of options for ACLs in zfs you probably want turned on , acltype and aclmode, and xattr=sa. Also case sensitivity and the case normalization stuff. And atime=off. Does anyone ever actually use access time because I can’t think of any use case for that on my nas!?
To go beyond chmod, you probably want to learn about setfacl and getfacl and set some sane ACLs so your folders and files are created with the correct permissions. If you are sharing stuff with other users, this becomes a problem real quick unless you’ve got the ACLs set right and things get created with 775/664 perms. I like to also set chmod g+s so the default group of anything is the group of the parent, not the current users primary group.
Lastly, if you’re on zfs you can setup something like sanoid to automate snapshots. I just did that, hopefully it will save me next time I run rm -rf against the wrong folder by accident!
Not just added complexity, I'd ask: Where is the performance gain? How likely is it on a home NAS you're ever going to hit the L2ARC, even if it is tuned to continually update and wear itself out? Very low.
> Lastly, if you’re on zfs you can setup something like sanoid to automate snapshots. I just did that, hopefully it will save me next time I run rm -rf against the wrong folder by accident!
PSA: sanoid is great and everyone should upgrade their experience of sanoid with httm[0].
Apple support is less than helpful.
I've recently built a small Home-Server with an old Fujitsu D3417-B2 Board, 32GB ECC RAM and Xeon E3 1225v5, 2TB Samsung Pro and Fractal Define Mini C for about 400 Bucks total (used) and can now enjoy fully encrypted ZFS Proxmox with LXC containers and Device Passthrough running 9.4W Idle.
I did not expect this to work that well, but I'm pretty happy with this config and using zfs-auto-snapshot from xshock[1] and OpenMediaVault LXC container from tteck[2] just for shares I don't think I would switch to a pure NAS again.
You have to teach yourself some zfs command line kung fu, though.
[1]: https://github.com/extremeshok/xshok-proxmox [2]: https://github.com/tteck/Proxmox
zfs-auto-snapshot is cool, but it's all in how you use those snapshots? httm [0].
But httm looks cool, I'll definetely take a look, thank you very much.
Not true for mirror vdev, but you don’t get the added capacity. You may add vdevs later, however.
Also some good arguments are made for using mirror vdevs over RAID-Z [1]. The extra space afforded by a 1:3 RAIDZ compared to 1:2 mirror isn’t much. Given the performance gain over a mirror resilvering and reduced recovery time, I prefer mirroring.
[1] https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs...
No idea if this has filtered through to mainstream release yet, but it was announced almost 18 months ago.
(Edit — AFAICT it is still in the pipeline but not yet quite released)
What was announced?
Here we are - https://arstechnica.com/gadgets/2021/06/raidz-expansion-code...
I ended up freeing up that SATA port I needed, but I'm waiting just a bit longer to expand ZFS on to that last drive so this code can get juuusstt a bit more testing!
To build a traditional RAID10 on ZFS, you'd create several mirror vdev's, then add them to the same pool. This writes one strip to vdev1, the next to vdev2, etc.
Should also be noted that it doesn't rebalance, so adding a vdev to a pool won't re-distribute data throughout the pool; new data will concentrate on the new vdev and potentially degrade performance until they're roughly at the same fill level.
(edit) Real world example: this is a NAS that has 3 pairs of mirrors. Writes are a little faster than a single disk since it can hit three at a time. Data correction is possible because every chunk of data has a mirror. And, since it's three mirrors I could potentially lose 1 disk in any vdev at the same time without data loss.
% zpool status
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0 ONLINE 0 0 0
ada2 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
ada3 ONLINE 0 0 0
da0 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0Are there any Wireless Storage solution that doesn't use the "Network", i.e Behind a Router. If not, are there any NAS solution that only works for "Intranet" or a Single User only.
Basically I want an external storage solution that doesn't require a cable. And offer ZFS / BTRFS error protection. And it has to be consumer friendly, i.e Plug and Play.
I want to know how to calculate expected performance, understand the process for disk monitoring, what to do in the event of a failure (my guess is to not power down the cluster in case disks don’t spin back up), etc.
Are you using WiFi or wired network connectivity using OMV?
Feel to reach out via email if you need help (see my profile...).
---
Using Samba instead of AFP (recommended in 2021) As pointed out by many people on Reddit that Apple has effectively deprecated AFP, I would like to add this section telling you how to use Samba instead of AFP for Time Machine and file sharing.
---