Building a Budget Homelab NAS Server
mtlynch.io
mtlynch.io
I wonder what he means by this. If he's referring to SHR, then it's just standard mdraid and Synology themselves have instructions on how to mount the volume in Ubuntu https://kb.synology.com/en-us/DSM/tutorial/How_can_I_recover...
edit: He later mentions encrypted volumes but those are also just using standard encryptfs https://www.impedancemismatch.io/posts/decrypt-synology-back...
This is one of the reasons I feel comfortable recommending Synology devices - there's not a lot of lock-in
- ZFS is pretty amazing in it's abilities, with it ushering in the age of software RAID over hardware RAID
- ZFS shouldn't be limited to FreeBSD. The Linux port has come quite a long way. I'd advise you to use PPA over repo though, as many key features are missing from the version on repos.
- TrueNAS is more targeted towards enterprise applications. If you want good utility as a home user then give Proxmox or the like a look. Then you can make it into more than just a NAS (if you're open to it).
- If you want to make things even more simple then consider something like UnRAID.
- ZFS' snapshotting can really shine on a virtualization server application, with the ability to revert KVM VMs to a previous state in a matter of seconds. Lookup Jim Salter's (great dude) Sanoid project to see a prime example.
- I don't recall why, but I've heard that RAIDZ should be avoided, in favor of stripped mirrors.
I would advise using ZFS only with distros that come with it (i.e. Ubuntu, Proxmox), especially if you plan to have your / on it. I wasted too much time on CentOS with ZFS, would not do it again.
I would also say Ubuntu is probably the better choice for Linux ZFS, as CentOS seems to be lacking good support.
I just didn't want to mention it because the discussion was mainly about Linux. But FreeBSD has a really strong toolchain for this indeed.
Want to use the last version of your firewall config? I wrote a utility you might like to try, httm[1], which allows you to restore from your snapshot-ed unique versions.
If you like ZFS, then trust me you have to have ZFS on root.
Zero setup, works out of box. Highly recommend ZFS and Ubuntu with ZFS!
If OS doesn’t boot, you boot from the latest snapshot! Every time you run apt-get upgrade, a system snapshots is taken automatically and an entry is added to boot menu.
The update "helpfully" updated initramfs for older kernels too... and if something broke, it broke previous versions too, so they all were unbootable. Eventually I ended up with an USB stick at hand with known bootable environment :(
The boot corruption could occur with the default file system ext4 also, except with ext4 there l is no recourse.
Needless to say, you can always boot from a live USB and mount your ZFS pool (and perhaps roll back).
See: https://openzfs.github.io/openzfs-docs/Getting%20Started/Ubu...
Incidentally its also pretty great because no business buys them second hand without warranty. So they're usually available for half nothing.
I don't use raid cards right now but I do use fibre channel which is also dirt cheap second hand
With TrueNAS I can move my drives to any other computer with the right interface and they will just work. I did this in the past 10 years of using TrueNAS.
If the manufacturer is late they can blame them, after all your IT manager paid for 4hr support so they've covered their ass.
If your TrueNAS fails at work, it's your ass on the line :P
I totally agree these special drive formats are really annoying. In this case I'd probably keep a spare on hand myself.
I call this sort of thing a technical guarantee rather than a commercial guarantee
> This is in fact one of the very few places where Hardware RAID solutions can have an edge over Software solutions - if you use a hardware RAID card, the extra write copies of the data will not have to go over the PCI bus, since it is the RAID controller that will generate the extra copy.
I was intending to use these disks for local backup and for storing rips of my extensive CD and DVD collection. As sibling comments mention, the possibility of the hardware controller failing is a worry, so I’d need to have a backup strategy for the backup disks. Since it’s going to be a home server, down-time wouldn’t be a problem.
I don’t have much experience with either hardware or software RAID so I’d welcome any advice.
¹ https://raid.wiki.kernel.org/index.php/Overview#What_is_RAID...
Don't take that too much into consideration - the article was last updated in 2007 ( https://raid.wiki.kernel.org/index.php?title=Overview&action... ) so it lacks some details (the same can be said as well for many ZFS-related infos that you might find) => nowadays doublechecking articles related to raid and ZFS is a must.
In my case I bought some HBA (Host Bus Adapter) cards (e.g. LSI SAS 9211-8i), set their BIOS to not do anything special with the HDDs connected to it (to be able to use them as well with other controllers) and used mdadm (earlier) or ZFS (now) to create my RAIDs => it works well, i get max throughput of ~200MiB per disk, I have all fancy features of ZFS without the problem of proprietary stuff related to the controller card :)
This is not a thing to bother (especially after we moved from PCI to PCI-E) for a home user.
The only great thing about HW RAID is what in case your primary drive fail-but-not-fail-completely, ie it would still be seen in the BIOS and BIOS would try to boot from it (but it wouldn't be able to, because drive is half-dead) is what for the BIOS a controller presents a single device and so it would allow booting from a healthy drive.
But again, if this is not a server in a remote oil digging site served twice a year by air (been there, done that) this is not a thing to bother for a home user.
> the possibility of the hardware controller failing is a worry, so I’d need to have a backup strategy for the backup disks
If you use a basic mirror (striped or not) the recovery process is straightforward - for a simple mirror just stick it in any other system/controller, for a striped you would need GetDataBack or R-Studio or just find a newer model of the RAID card from the same vendor.
In your case I would advise to have a single disk in the ODD bay as a boot/system drive and use both your HDDs as an LVM PVs, without fdisk shenanigans. If you/when you decide to upgrade/replace disks the migration would be just a couple of commands like
pvcreate /dev/sdc
vgextend your_vg /dev/sdc
pvmove /dev/mapper/your_vg__your_lv /dev/sda /dev/sdc
vgreduce /dev/sdaThanks for the great advice on using LVM and the commands for replacing a disk. I only recently cane across a system that used the disk block device itself as a physical volume – rather than partitioning it via fdisk or parted. Other than using LVM for creating snapshots, I haven't really used it for anything interesting.
The array metadata is just stored at the end of the disk, so this is not a problem to just attach a disk somewhere rlse
> commands for replacing a disk
Take it with a grain of salt, I just wrote them from memory. But the overall process is exactly as I said.
Glad you found that helpful.
Some notes.
- Not having Raid really doesn't matter. The primary purpose seems to be to save a little bit of space by clever checksumming or increase read performance from parallel operation, but none of this is valuable to me.
- I use ext4. I think it would make sense to move to a snapshot-capable system to make the periodic rsync-backups more correct (I don't even bother dropping to r/o mode, since it's fairly static data).
- What really keeps me up at night: bit flips silently corrupting files. I think btrfs or ZFS are supposed to solve this through constant background checksumming. I really need a periodic process to checksum every file and take action on exceptions. Note that RAID will not help you with this.
- This has worked pretty well so far. Twice already (over 12 years) have I had one disk in the pair fail, upon which time I would order a new pair (they were bigger/cheaper by that point and I figured the other one would fail soon) and rebuild the server.
The consensus seems to be that RAID isn’t particularly useful for a home server. Backup would be more useful as I would be more likely to accidentally delete or over-write a file than for a disk to fail catastrophically) so I think I might use the second disk, similar to how you use yours. I would also be better served by using ZFS; its de-duplication would also be useful – so I’m going to try it as an experiment.
Agreed. Also for anyone using NixOS, I've found its ZFS support is first class and easy to set up:
https://www.reddit.com/r/NixOS/comments/ops0n0/big_shoutout_...
Most people care about random IO (also once your filesystem has been populated and in use for a while, true linear IO really ceases to be due to fragmentation.) Striped arrays lose random IO performance as drive count goes up; an array of mirrored pairs gains random IO performance. This is less of an issue with tiered storage and cache devices, especially given you almost have to work to find an SSD less than 256GB these days.
You can only upgrade a zdev by upgrading all its drives; it's a lot nicer cash-flow-wise to gradually upgrade a mirrored pair here and there, or upgrade exactly how many pairs you need to for the space you need.
With RAID-Z you have a drive fail and pray a second doesn't fail during the resilver. With RAID-Z2 you can have any two drives fail. With mirrors you can lose 50% of your drives (provided that they're the right drives.)
Enough concurrent clients doing sequential IO also looks like random IO to a storage server.
This is no longer the case, or at least, should no longer be the case soon. The ability to add drives to a zpool has been announced, and will trickle through to stable before too long.
FreeBSD migrated from own ZFS to OpenZFS so you have single ZFS implementation in BSD and Linux https://openzfs.github.io/openzfs-docs/Getting%20Started/Fre...
Raidz needs to read all of every drive to rebuild after a drive replacement while a striped mirror only needs to read one. However if you're regularly scrubbing zfs then you read it all regularly anyway.
Raidz effectively has a single spindle for random or concurrent I/O since a whole stripe needs to be read or written at a time. Raidz also had a certain amount of wastage owing to how stripes round out (it depends on how many disks are in the array), but you still get a lot more space than striped mirrors.
For a home user on a budget raidz2 usually makes more sense IMO, unless you need more concurrent & random I/O, in which case you should probably build and benchmark different configurations.
I've been using zfs for over 10 years, starting with Nexenta, a defunct oddity with Solaris kernel and Ubuntu userland. These days I use Zfs on Linux. I've never lost data since I started.
Not quite. Each vdev rebuild uses disks within it. A pool with multiple vdev each being raidz does not need to read all disks to rebuild a single raidz vdev. Your statement compares one vdev vs many vdevs. It just happens to be the case folks assume one large vdev with raidz and multiple vdev with mirroring.
If you have a 3- or 4- way mirror wouldn’t ZFS read from all disks in the vdev to rebuild any added disks to the mirror (there can be more than one)?
Well if you're aiming for a specific ratio but want greater capacity without upgrading all disks to larger capacities, your only option is to use more vdevs configured just the same.
Example: 66% usable capacity, 6 disks in a raidz2 (4 usable + 2 parity) or 9 disks in a raidz3 (6 usable + 3 parity). If you want to add capacity but maintain your parity ratio (for a given fault tolerance risk) there is no raidzN with N > 3, so you must add vdev.
Increasing the size of the raidz vdev means you're reducing your failure tolerance.
I feel SSDs are in a totally different category that makes Z1 an actual option, whereas I don't trust it for spinners. Key difference being that a failed SSD can usually be read (only) whereas a failed spinner is usually as good as bricked.
It's been great, right up until one of the sticks of RAM started to fail...
I have questions about this. I'm thinking of building my own NAS server, and I don't know which OS to use. On the one hand it looks like people recommend TrueNAS a lot, which is nice now that they have a Linux version, but I'm not really sure what does it offer over a raw Debian apart from web/configuration and some extra tools? I have quite some experience in running Debian systems and managing RAIDs (not with ZFS but doesn't seem too much of a jump) and I worry that TrueNAS, while nice at the beginning, might end up being limiting if I start to tweak too much (I plan on using that NAS for more things than just storage).
Also, TrueNAS makes setup painless: users, permissions, shares, vdevs, ZFS tuning, nice dashboard etc. With Debian, you get a lot of config files and ansible playbooks that become hard to manage.
Ideally you won’t run other stuff on a NAS, outside Docker.
I will definitely argue that TrueNAS gives stability and ease of management. Some of that can be found with Proxmox too though. I think it just really depends on which medium you prefer. Perhaps trying both is the best option?
Isn't proxmox just virtualization? didn't know it can be used as a NAS too.
I consider myself an intermediate homelabber and a TrueNAS beginner. I just built my first NAS server, so wrote this to capture everything I wish I'd known at the start. I hope it's helpful for anyone else thinking about building their first NAS server.
Any questions or feedback about the post are more than welcome.
Some suggestions for anyone else looking to do the same:
i3 runs a bit cooler than ryzen, still 8 threads. 8tb WD blues (they're SMR at 8 and up). You can find Atx boards with 8 sata ports and dual nvme slots for caching / fast pools.
SMRs are a fucking blight.
RAID-Z1 is something I never consider without a solid backup to restore from and a plan to execute that process at least once in the lifecycle of an array.
If you suffer a total disk failure of one of those disks in the array, you have likely lost some data. The good news is that ZFS will tell you exactly which files you have lost data for and cannot rebuild. If you have those files, you can overwrite them with the backups to get your integrity back.
The reason is, with a total loss of a single disk, any read error on any of the remaining disks is a lost/corrupted file.
For this reason, you need a strong(easily accessible, consistent, current) backup strategy and an acceptance of downtime with Z1.
As for ECC, it's better, but your absolute worse case scenario is that you get a bit flip before the sync and hash happens, and now that bit flipped data is committed to disk and you think it's OK. I prefer ECC to avoid this, but you are still reaping a multitude of benefits from ZFS without ECC.
The only valid rule for RAM and ZFS is that more RAM = more caching of recently read data. Single, or very few user appliances will see little benefit past 8GB even with 100TB unless you happen to be reading the same data over and over. Where ZFS shines is having hundreds of gigabytes of RAM and tens or more concurrent users mostly accessing the same data. That way the vast majority of reads are from RAM and the overall disk IOPS remain mostly idle.
Most of the ZFS RAM myths come from Deduplication, which should be disregarded as a ZFS feature until they allow storing of the DDT on a Optane-like latency device. Even better would be offline deduplication, but I doubt that will be a thing in ZFS this decade.
Wait, what? If a RAID-(z)1 ZFS array loses one disk, there's data loss? I've ran so many RAID-1 and RAID-10 arrays with mdadm that I can't even being to count them, and I had many drive failures. If any of those arrays would have corrupted data, I would have been mad as hell.
What I am missing here? How is this even remotely acceptable?
That is the meat of it. With traditional RAID it is the same issue, except you never know it happens because as long as the controller reads something, it's happy to replicate that corruption to the other disks. At least with ZFS, you know exactly what was corrupted and can fix it, with traditional RAID you won't know it happened at all until you one day notice a corrupted file when you go to use it.
RAID-Z1 is better than traditional RAID-5 in pretty much every conceivable dimension, it just doesn't hide problems from you.
I have encountered this literal scenario where someone ran ZFS on top of a RAID-6(don't do this, use Z2 instead). Two failed drives, RAID-6 rebuilt and said everything was 100% good to go. A ZFS scrub revealed a few hundred corrupted files across 50TB of data. Overwrote the corrupted files from backups, re-scrubbed, file system was now clean.
ZFS automatically self-heals an inconsistent array (for example if one mirrored drive does not agree with the other, or if a parity drive disagrees with the data stripe.)
ZFS does not suffer data loss if you "suffer a total disk failure."
I have no idea where you're getting any of this from.
The poster built a (non redundant) zfs pool on top of a hardware raid6 device. The underlying hardware device had some failed drives, and when rebuilt, some of the underlying data was lost.
ZFS helped by detecting it instead of letting the bad data though like would normally have happened.
See eg here where the increasing disk size vs specified unrecoverable read error rate is explored in relation to the question at hand: https://queue.acm.org/detail.cfm?id=1670144 (in the article Adam Leventhal from Sun, the makers of ZFS, talks about the need for triple parity).
Also, the conclusion "ensure your backups are really working" is an important point irrespective of this question, since you'll also risk losing data due to buggy software, human errors, ransomware, etc.
In RAID-Z, you can lose one drive or have one drive with 'bit rot' (corruption of either the parity or data) and ZFS will still be able to return valid data (and in the case of bit rot, self-heal. ZFS "plays out" both scenarios, checking against the separate file checksum. If trusting one drive over another yields a valid checksum, it overwrites the untrusted drive's data.)
Regular RAID controllers cannot resolve a situation where on-disk data doesn't match parity because there's no way to tell which is correct: the data or parity.
thanks for the clarification.
in that sense, yes, of course, if you have bit rot and another disk failing, things go south with just two disk. ZFS is not magic.
How does Z1 recover the data in this case other than alerting you of which files it cannot repair so that you can overwrite them?
The drives have been champs overall, they're approaching an average runtime of about 8 years. During that 8 years we've lost about 20% of the drives in various ways.
It is almost guaranteed that when a drive fails, another drive will have a URE during the resilver process. This is a non-issue as we run RAID-Z3 with multiple online hotspares.
Are they used 24/7 at high iops? Why not nightly scrub?
I intentionally provisioned one of the long term archive only appliances with 12 hot spares. This was to prevent the need for a site visit again before we lifecycle the appliance. Currently down to seven hot spares.
That replacement will probably happen later this year. Should reduce the colo cost by power requirement reduction enough that the replacement 200TB appliance pays for itself in 18 months.
- for ZIL to do its work properly, you need the disks not to lie when they claim that the data has been truly saved. This can be tricky to check, so perhaps think about a UPS
- if you have two M.2 slots you could use them to mirror two partitions from two different disks for your data pool's SLOG. The same could be done to form a new mirrored ZFS pool for the OS. In my case I even prefer the performance that a single-copy SLOG gives me at the risk of losing the most recent data before it's moved from the SLOG to the pool.
or your house could burn down
or somebody could steal the computer while you're away on vacation
or lightning could strike your electrical grid service entrance or a nearby pole/transformer, causing catastrophic damage
or your house could flood
lots of other things.. if you really have important data it's important to plan to for the total destruction of the storage media and server holding it.
Not really. You need to be synchronizing to a _write-only_ backup archive. A local ZFS snapshot can be deleted locally.
(Also fire, compromise, police confiscation, etc.)
For my write-only backup needs, this free service known as S4 works wonderfully:
http://www.supersimplestorageservice.com/
(This is a joke.)
That's because of overhead in TCP over IPv4. You're testing the payload throughput, not the physical throughput. The theoretical maximum performance without jumbo frames is around 95%.
https://en.wikipedia.org/wiki/Jumbo_frame#Bandwidth_efficien...
- with same disk size, but just 3 disks, I get around 240 MB/sec read speed for large files (with 10 Gbps NIC). I guess the biggest difference is the CPU power, your NAS seems very slow. On 1 Gbps NIC I get 120 MB/sec transfer speed. My system is even virtualized, on bare metal may be a little bit faster.
- you cannot expand your pool, if you add one more disk there is no way to cleanly migrate to a 5 disk raidz1. There is some new development that kind of does something, but it is not what is needed
- unless esthetics is a big deal for you, there are still $30 cases around. The extra $70 can be used for something else *
- * with a small percentage cost increase, an investment in CPU and RAM can give you the capability to run some VMs on that hardware, so that CPU will not sit at idle 99.9% of the time and be underpowered when you do use it. Using a dedicated computer just for a NAS is not very cost and power efficient, but if you group multiple functionalities it becomes a great tool. For example I run 3-4 VMs at all times, up to ~ 12 when I need it.
- that motherboard and the comparison to a B450 is wrong. The MB restricts you to 4 SATA, while the B450 I bought for ~ $120 has 6 SATA ports
- TrueNAS does not *require* a HBA firmware change, that is needed if you want to convert a RAID controller to plain HBA mode or with certain old HBA that need newer firmware. However for your setup a HBA is not needed. If you want to add many disks and have a good performance (like more than 500-1000 MB/sec) then you need the HBA
- your math is wrong. You calculate available space using ~ 3.8TB disks and divide to 4 TB. The 4TB disks don't have 4TB, but 4x10^12 bytes, so the percentages in your table are exactly 80%, 60% and 40%.
- that CPU does not work with 32GB DIMMs. This works only with newer Ryzen generations, not with Zen+ in this CPU.
- GPU is not missing. TrueNAS does not render anything on a GPU, there is no need for one. I did ran TrueNAS for a couple of years on a computer with no video capability at all (a Ryzen 2700) without any problem, I just used a GPU for the initial installation and then removed it.
- unless you store a database for a SQL server or similar, there is no benefit in a SLOG; it is not a tiered cache, so it does not speed up file transfers in any way. You can have a disk dedicated as a read cache, but the cache content is currently wiped at every restart (a documented limitation) and not needed if you don't want very good performance with small files over the network
And if you have just a few powerful workstation desktop PCs it's also worth it to connect them at 10GbE to a new switch.
here's a fairly typical one. these have excellent freebsd and linux kernel driver support.
https://www.ebay.com/itm/265713815725?epid=1537630441&hash=i...
Then run IP over IB on each host and you have a 56 Gbit network that all your applications will just see as another network interface on each host.
if you get a $200 switch with a few 10GbE interfaces in it you can easily expand things in the future by trunking vlans to another newer 10GbE capable switch, or connecting to a switch that has multi-gig copper ports for access ports to 2.5/5GBaseT capable desktop PCs and laptops, etc.
$40 for a 10 meter fiber optic cable is a high price when you can buy LC-LC UPC 9/125 duplex 2 meter cables for $3.50 to $4.70 a piece (or a few cents more for additional meters) and connect them between $20 transceivers. no matter what route someone goes with would recommend buying $30-40 of basic fiber connector cleaning supplies.
https://www.fs.com/products/40192.html?attribute=193&id=3026...
if one wants to buy used weird previous gen dead-end stuff there are also tons of very cheap 40GbE mellanox ethernet adapters on ebay with the QSFP to go with them, and if you have a place to put a switch that doesn't matter if it's noisy like in a wiring closet somewhere, cheap 1U switches with 40GbE ethernet ports on them that can also be used as individual 10GbE when broken out.
I'll just clarify that I meant you can get a 10m fiber optic cable with two transceivers for $40.
A NAS home build of this size will never exceed 10 Gbps, I barely get ~ 2 Gbps out of the spinning disks.
for instance I have a setup which is meant for working with uncompressed raw yuv420 or yuv422p 1080p and 4K video, there's a 512GB NVME SSD and a 1TB SSD set up as individual JBOD and exposed to the network for video editing scratch file storage, and it will definitely saturate 10GbE.
this is actually needlessly complicated, if/when I build a more powerful desktop pc again I'm just going to put the same work file space storage on a 2TB NVME SSD directly stuck into the motherboard.
As an example here is the driver page for Mellanox, now owned by Nvidia, since they are a major Infiniband equipment supplier: https://network.nvidia.com/products/infiniband-drivers/linux...
It seems that some decent support only exists for more recent generations. The older ones like ConnectX-3 or earlier, which typically show up on ebay are either not supported any more or maybe available for older kernel versions and soon to be EOLed.
So do I understand it correctly that to use such adapters one has to actually downgrade to an older kernel version?
Or is there some basic support in the latest Linux kernels for older generations still?
I've not actually used IPoIB on such gear myself, but we have been working quite a bit on reusing old/ancient HPC clusters with IB adapters, and you can generally make things work if you spend enough time on trial and error and you are not afraid of compiling code with complicated dependencies. As long as you can get the IB stuff talking, and the driver is using OFED, the IPoIB part should Just Work.
It is always going to be an adventure working with used gear. But HPC has such a high decommissioning tempo and low resale value that there will always be quite a few other enthusiasts toying about.
Energy use is very low.
Yeah, I've never thought much about power consumption, but I've done a few write-ups of previous builds, and I received a lot of questions about power draw, so I decided to measure it on this one. I was surprised at how much power the system consumed, and it will be something I think about more up-front on future builds.
If you are not after speed, then you can do a redundant array of cheap nodes. Instead of using raid, just shove in an 8-12tb disk in a number of thin clients.
The key is that they spend most of the time turned off.
https://pcpartpicker.com/list/dLhNvf
I used a Supermicro MB and ECC RAM. It's not much more expensive and it's nice having IPMI. I personally think it's crazy to forego ECC. The SAS controller, expander, and drives were used, everything else was new. Prices have gone up. The new parts were $638 at the time. The drives were ~$20/ea. The HBA and expander were ~$85 for both. After the fans, cables, extra drive cage and brackets, total cost was around $1K. This hardware is all supported out-of-the-box by TrueNAS. I haven't done the math to figure out when the cost of running this will exceed having purchased higher capacity drives.
This is what a typical used SAS drive looks like. 30K hours hours but very few on/off cycles. Zero defect list:
The failed SMART test turned out to be a firmware issue. I had to update the firmware on all the drives. That was a bit of an adventure:
A few drives arrived with a non-zero defect list or otherwise failed burn-in. I contacted the seller on eBay and they sent me replacements w/o any fuss. I'm not necessarily recommending used SAS drives, but I'm not recommended against them either. I will recommend the serverbuilds forum for generally good advice on all this. I think this post got me started:
https://forums.serverbuilds.net/t/another-nas-killer-4-0-bui...
The current NAS killer version is 5.0:
*Clean in terms of dust, not cable management
I'm sad you don't think much of the cable management. I think I did pretty good considering there's 11 drives, 6 fans, and I wasn't able to use custom lengths on any of the cables.
The ease at which you can revert mistakes using ZFS snapshots is much better compared to restic. You can pretty much navigate to the correct snapshot on your live filesystem and restore whatever you need to restore.
It also makes backups easier as you can just send the snapshots to the backup device (another server or external storage device).
rm -rf tmp *
rm -rf tmp*Yea, i love up 1 directory before I delete anything.
It took me a minute, but I assume this should be "move up". This seems like a good habit.
Having NFSv4 ACL access is a huge plus since you can configure permissions natively from windows and have them enforced even on the shell.
Storing them in metadata is not the same as having them natively.
$ synoacltool -get .config
ACL version: 1
Archive: is_inherit,is_support_ACL
Owner: [semiotic(user)]
---------------------
[0] user:semiotic:allow:rwxp-DaARWc--:fd-- (level:1)
[1] group:users:allow:r-x---a-R-c--:fd-- (level:1)The ACLs do work via NFS and it also works with Active Directory. They ship an AD implementation too, if you are interested in that (it is actually Samba in AD mode).
> I chose raidz1. With only a handful of disks, the odds of two drives failing simultaneously is fairly low.
Which is not really the case of you bought x amount of the same disks and always use them together. I had that happen to me just a few months ago. 4 identical discs bought at the same time. Raidz1 reported one dead/dying disk so I replaced it and started resilvering, which can take days and leaves the disks at 100% utilization.
So after 12 hours or so a second one failed and the data was gone.
Lesson learned: mix up your disks
I bought 4x Seagate Ironwolf Pro 12TB drives from different vendors, one failed after a year, then when I got the replacement another drive failed during the rebuild, and then 6 months later the replacement failed. Now another one of the original drives is also reporting reallocated sectors.
Same system has 4x WD Red drives which have been running fine with 0 reallocated sectors for almost 7 years.
I'm in the same boat. I configured remote backup systems on a handful of computers. I think I reached for backups only twice over the last ten years. Of course I need something, backups or snapshots, but for my use case snapshots (with a network copy) would need work to set up. And if the remote storage is worse, that would be more of a problem than the changes in the restore process.
Having a large NAS has had an interesting (though predictable) impact on all the computers around it: Pretty much every bit of data lives on the NAS (accessed either by CIFS, NFS, or iSCSI). When I had to reinstall Windows, it was mostly painless because all my important data and Steam games library was on a remote iSCSI disk. When I replaced the drives on my linux servers, I didn't have to backup hardly anything, as I worked almost exclusivly on NFS-mounted directories. When bringing up a new raspberry pi for projects, it also has instant access to more terabytes of storage than it could ever need.
Also, for a homelab, getting 10GBe fiber between two machines is surpringly cheap and easy. For certain workloads, it can be a noticable speed boost over 1GBe.
I ask because the difference between a SSD and a hard drive can be massive in this regards, so I'd be really interested to know if the network latency is also a comparable hit.
My gaming computer had an old SATA SSD (Samsung 840 Evo IIRC). Some games took ages to load (particularly Fallout 4). I switched to a much faster NVME drive, and subjectively, it's not any faster loading games. I'd say this was a very underwhelming purchase.
For reference here's the drives I have. I don't think they're by any means the loudest and nor are they the quietest. The perception of noise can be a lot to do with surroundings, if you have wooden floor instead of carpet and a whole number of factors. For me the constant whiring is too much.
https://www.amazon.co.uk/gp/product/B07H289S79/ref=ppx_yo_dt...
There were some 1U products which included RAID support, priced around $500, which is a bit much for 1U chassis + SATA/SAS backplane + Pico power supply. 1U chassis with ~11" depth (seems to be a telco standard?) start around $100.
StarTech 1U JBOD discontinued, https://www.startech.com/en-us/hdd/sat35401u
Silverstone RS431 JBOD unavailable, https://www.silverstonetek.com/product.php?pid=482&area=en
iStarUSA upcoming (shipping in June) 1U JBOD is $400, http://www.scsi4me.com/istarusa-m-140ss-jb-1u-3-5-4-bay-tray...
For ~$600, QNAP has an Arm-based 1U short-depth NAS with 2x10GbE and 2x2.5GbE networking, plus dual M.2 NVME slots. Maybe Armbian will run on that SoC, it's part of a supported family. https://www.qnap.com/en-us/product/ts-435xeu
$100 ODROID M1 SBC has an M.2 NVME slot with 4x PCIe lanes. In theory, this could be bridged to a PCI slot + LSI HBA, within a small case, as a DIY low-power NAS.
However I do not recommend his choice of 4 x 8TB drives in a raidz1. Financially and technically it doesn't make sense. He spent $733 for 24TB usable ($30.5/TB)
He should have bought fewer, larger drives. For example 14TB drives sell for $240. So a config with 3 x 14TB in a raidz1 would total $720 for 28TB usable ($25.7/TB). Smaller costs, more storage, one less drive (= increased reliability)! It's win-win-win.
Especially given his goal and hope is in a couple years to be able to add an extra drive and reshape the raidz1 to gain usable space, then a 14TB drive then will be significantly cheaper per TB than an 8TB drive (today they are about the same cost per TB).
Actually, with only 8.5TB of data to store presently, if I were him I would probably go one step further and go with a simple zfs mirror of 2 x 18TB drives. At $320 per drive that's only $640 total for 18TB usable ($35.6/TB). It's a slightly higher cost per TB (+17%), but the reliability is much improved as we have only 2 drives instead of 4, so totally worth it in my eyes. And bonus: in a few years he can swap them out with 2 bigger-capacity drives, and ZFS already supports resizing mirrors.
Where? Also, is it worthwhile to buy hard drives explicitly for NAS when you're using ZFS? For example, Seagate has the IronWolf product line explicitly for NAS and cost more.
Drives branded for NAS applications differ slightly from mainstream drives. For example Seagate claims the IronWolf is "designed to reduce vibration, accelerate error recovery and control power consumption" which essentially means the drive head actuators will be operated more gently (reduced vibration) which slightly increases latency and slightly reduces power consumption, and also the firmware is configured so that it does fewer retries on I/O errors, so the disk commands time out more quickly in order to pass the error more quickly to the RAID/ZFS layer (why wait a minute of hardware retries when the RAID can just rebuild the sector from parity or mirror disks.) IMHO for home use, none of this is important. Vibration is only an issue in flimsy chassis, or extreme situations like dozens of disks packed tightly together, or extreme noise as found in a dense data center (see the video of a Sun employee shouting at a server). And whether you have to wait a few seconds vs a few minutes for an I/O operation to timeout when a disk starts failing is completely unimportant in a non-business critical environment like a a home NAS.
A 500W PSU won't necessarily draw more than a 250W PSU, that is merely its maximum sustained load (what the rest of the system asks for) rating. The Bronze 80+ rating is likely part of the problem here, that indicates what the power draw from the wall is compared to what is being provided to your system. Titanium 80+ would net you about 10% reduction in wall power usage. Keep in mind that manufacturers play fast and loose with the certification process and a consumer unit may not actually be what it says on the box, you need to rely on quantitative reviews.
Other than that, spend some time in the firmware settings. Powtop also does a great job at shaving off some watts.
Mostly true, but not exactly. Most computer PSUs are more efficient when operating around 50% of their rated load. So if a computer consumes 125W internally, a 250W PSU would translate to lower power consumption measured at the wall than a 500W PSU, typically by about 2-5%.
For example see the chart https://www.sunpower-uk.com/files/2014/07/What-is-Effciency.... (115 VAC input) : 88% efficiency at 25% load, vs 90.5% efficiency at 50% load. In practice if the consumption is 125W at the PSU's DC output, this translates respectively to 142W vs 138W measured at the wall.
This 2-5% difference may not seem much, but it's similar to upgrading 1 or 2 levels in the 80 PLUS ratings (Bronze, to Silver, to Gold, to Platinum, to Titanium).
A 250W 80-bronze PSU for a 60W load will be operating at 25% capacity and 82% efficiency or better.
A 500W 80-titanium PSU at 60W will be at around 12% and 90% efficiency or better.
So, an 8% difference in minimum required efficiency...for a huge increase in cost.
It's much better to buy a high "tier" PSU (for reliability and safety), sized so that it spends most of its time at or above 20% duty cycle (which in OP's case would indeed be 250W.)
80-gold is very common in the marketplace and where most people should probably be buying.
- https://vermaden.wordpress.com/2019/04/03/silent-fanless-fre...
Only if you buy different hard drives or at least from different production batches. I had a lot of trouble on the same premise and I won't make that mistake again.
Edit: He mentioned it though ( a bit later in the article)
> The problem is that disks aren’t statistically independent. If one disk fails, its neighbor has a substantially higher risk of dying. This is especially true if the disks are the same model, from the same manufacturing batch, and processed the same workloads. Given this, I did what I could to reduce the risk of concurrent disk failures.
> I chose two different models of disk from two different manufacturers. To reduce the chances of getting disks from the same manufacturing batch, I bought them from different vendors. I can’t say how much this matters, but it didn’t increase costs significantly, so why not?
Please elaborate, I'd love to hear your story!
I hear a lot of advice around raid/z-levels and it often seems backed up by shaky math that doesn't seem to be backed up by reality (like the blog posts that claim that a rebuild of an array of 8 TB drives will absolutely have hard read errors, no exceptions, and yet monthly ZFS scrubs pass with flying colors?)
Not much to tell otherwise though. I was really annoyed with it at the time.
* https://www.techradar.com/news/larger-than-30tb-hard-drives-...
Crazy.
The problem is when the second drive fails while you’re recomputing the parity after having replaced the first faulty drive, a process which may stress the discs more/differently than regular operation, and also tends to take some time. Raidz 2 (or Raid 6) helps to provide some redundancy during that process. Otherwise you don’t have any until the Raid has been rebuilt.
If you plan to build a budget NAS and enough room is not a problem, I personally would recommend to get an old and used Dell T20 Xeon E3-1225v3 with min. 16GB ECC DDR3 RAM, 2x10TB Seagate Exos ZFS RAID and a bootable USB Stick with TrueNAS or if you prefer Linux, TrueNAS Scale / OpenMediaVault.
If room IS a problem, you could get a HP Microserver Gen8 or higher with a Xeon the config above.
- Server Cost: 150 Bucks
- Total Cost: 650 Bucks (150 for server, 500 for HDD)
- Power Consumption: 33W Idle, 60W Heavy File Transfer
- Silent enough without modding the fans
- Ethernet-Transfer-Speed: 110MB/s on 30% System Load
I do not own a 10Gbit ethernet card, but I'm pretty sure, transfer speeds with 10Gbit would be acceptable, too.
I have recently built a truenas box with 2x4TB ssd's. But I think I will want to expand it. Currently it runs at 14W idle. If I add 2hdd I expect it to increase to 40W(?). How can I optimize this?
BUT: You should always consider, that optimizing Power Consumption would mean to buy new hardware. I hardly ever do this. Buying new hardware in many cases is vastly more expensive than keeping your old hw, even if it consumes more power.
I'll try to give you an example:
- My Dell T20 consumes 33W Idle
- There is HW out there, that consumes 14W - diff (19W)
- Let's say my Dell costs 150$ vs 600$ (for new hardware) - diff: 450$
- How long can I run my Dell T20 for 450$, without investing in new hardware
Calculation:
- Dell additional power per year: 0,019kW * 24h * 365 = 166kWh
- Estimated cost per year (in germany): 166kWh * 0,30$ = 49,80$
- Means I can run it about 9 years to the break even point
Advice: Buy a newer cheap used server consuming 12W in 5 years, and you save a ton of money. AND buying used hardware is better for the environment...
Background about my situation: I've only ever used laptops and Raspberry Pis in my life. Currently, my home server is a couple of RPis but I've outgrown them and need to upgrade.
Point being, I've never built a server from scratch or even opened one up. I am planning to go with this for now: https://pcpartpicker.com/list/bzYZcb
What you are suggesting seems a better way for me to go about it. Can you share some links from Ebay or Amazon for the kind of servers I could get? Also, will these servers contain everything I need to get up and running (with the exception of hard drives, of course)?
(I am based in the US.)
I personally think, that the Dell T20 is more than enough for a ONLY NAS Server and even for a small homelab.
You could go for something similar to this (ebay) but I had MUCH more success in looking at "garage sale" websites (I don't know, what sites to use in the US):
https://www.ebay.com/itm/175291934211?hash=item28d036a603:g:...
https://www.ebay.com/itm/284831417242?hash=item425146839a:g:...
Dell T20 - but I would not invest more than 150$ incl. >= 16GB ECC RAM.
HP Microserver Gen8 is more expensive, but not more than 350$ incl. >= 16GB ECC RAM.
You could also consider FUJITSU PRIMERGY TX1310 M3 and better. More modern Machines include the DELL T40, but it is much more expensive.
Ha, should have mentioned that before! I want to run a few services at home, so need more than 2 GB RAM. I'll also create a NAS/Nextcloud to store "family" data (documents, pictures etc.) but nothing "data hoarder" level. So, 4 TB would easily last us a decade or more.
Do you still think I should go for such enterprise servers? (Noise, complexity etc.) Or should I just get a used desktop/laptop?
When I first bought the system over 10 years ago, ZFS on Linux wasn't really a thing, so I used FreeBSD. I later switched and with the switch came substantial power savings.
Lawrence Systems just ran benchmarks[0] between TrueNAS Core (FreeBSD) and TrueNAS Scale (Debian), but they didn't include power consumption, unfortunately.
I really like seeing other people's builds, but I know that building my own computer isn't something I want to do. I was happy to see the comparison between the DIY model and the roughly-equivalent commercial units. I'll likely buy another QNAP (to run TrueNAS on) when the time comes, and the comparison tells me that I won't get screwed too badly by doing so.
>I've gone the TrueNAS route, but I'm running it on a QNAP TS-451. I'm running TrueNAS off of a USB stick hanging off the back
Oh, I didn't realize that QNAP allows that. Synology makes it pretty hard to boot any other OS, and I assumed the other vendors were similar. I'll keep that in mind for my next build because I do have fun building servers, but I also really appreciate systems like Synology and QNAP where the hardware and case is optimized for the NAS use-case.
I use raid 6 and also backup my data externally to another nas as well as backup to a static usb drive. Backup requires multiple different types since failures are so catastrophic and can occur in ways you don’t expect.
ZFS can perform periodic scrubs to detect and repair bit rot, and I'm pretty sure TrueNAS is configured to do this by default
At some point I wanted to go the TrueNAS / FreeNAS / OwnCloud etc. route but after seeing the pages upon pages of troubleshooting and lost data horror stories I stuck with a commercial solution.
But Synology doesn’t use ZFS, which is a better filesystem than btrfs. In particular ZFS offers native encryption (instead of the clunky ecryptfs in synology), and allows ZFS send from Linux servers.
Plus, the storage is unlimited. Plus, it is more resistant to failures and disaster than anything home made. Plus, I don't have to store and take care of another noisy box in my home.
What area of the world do you live where you get 10 Gbps to the Internet? Can you reliably get 10 Gbps transfers to pCloud?
I got 1 Gbps fiber in the last year, but it's more like 700-800 Mbps in practice. I consider myself lucky to even get that, as my experience before that has always been 100-200 Mbps even on a "1 Gbps" plan. I'm super jealous of people who get a full 10 Gbps Internet connection.
Uploads to pCloud are about half of that while downloads can be over 1GB/s.
I run two offline Nas (I unpower them and do scrabing every month) and have one with raidz2 for all critical things like my photos.
To resilver 8tb takes ages and while he wrote his thoughts on it, I was missing the repair risk calculation
You never noticed also exactly because you can’t know about data corruption if you don’t run with ECC memory.
Either way, I've been using computers longer than TFA and wouldn't build a home lab box without ECC.
I prefer mixing brands/models instead. Two vendors _might_ get you a different batch, but you could be choosing a bad model. I ended up building mine from three different WD models and two Seagate ones. I'm paranoid and run with two spares.
One of the nice benefits of ZFS native encryption + s3backer is that if I had a total outage locally and needed to recover some files quickly I could mount the s3backer-based zpool from any machine, decrypt the dataset, and pull the individual files out of a filesystem. It's also a weird situation with cloud providers that convenient network-attached block storage is ~10X the price of object storage at the moment but performance can be similar using s3backer.
I will mention that I am one of those folks with a TS140. Love that it's a power sipper. I maxed out the processor and memory, as well as loading it up with two 10 TB rust disks and two 512 GB SSDs.
I do wonder what the power consumption figures would be though. His system was drawing an annoyingly large amount of power and I suspect that was mostly those HDDs.
Regarding the SLOG device, you probably don't need it for a file server, but if you do you can definitely free a drive bay for an HDD by just using double sided tape somewhere like on the PSU. I'm sure it's also possible to put three more HDDs above the CPU, right in front of the exhaust fan. If I had a 3D printer I would try to build something bringing the total to nine HDDs.
If you need more SATA ports but are running out of PCIe slots, you may be able to reuse an empty M.2 slot. An M.2 with two lanes of PCIe 3 gives you 5 SATA ports with an adapter[0].
Just make sure to get one that runs in IT mode or you have to mess with the firmware.
In case some people wonder what "IT mode" is, as I used to some years ago, what you basically want is a card that will expose the drives directly to the OS, as opposed to "volumes".
In other terms, if the card is a RAID controller, it may insist on you creating arrays and only expose those. You can circumvent it by creating single-drive arrays, but it's a pain.
Some cards can do both, but it's usually not advertised. Non-RAID cards also tend to be cheaper. Others (usually LSI) can be flashed with a non-RAID firmware, but again, it's less of a hassle to not have to do it.
The fancier more expensive ones are typically referred to as HBAs instead of "SATA cards" https://unraid-guides.com/2020/12/07/dont-ever-use-cheap-pci...
If you're doing this at home, you can get used enterprise gear on eBay (like an LSI SAS HBA) for the same price or cheaper than brand-new consumer gear, and it will probably still be more reliable (I built a 130 TB NAS for my friend's video production business and literally everything aside from the drives and the cables was bought used on online auction, and it's been humming along fine for a while now - the only part that was bad was one stick of RAM, but the ECC errors told me that before I even got around to running my tests on the sticks)
Since you are using an M.2 drive rather than a USB drive for your boot drive, you are not affected by the issue that affected me. But I've reached a point where I would not trust FreeBSD to not have weird and esoteric hardware issues that could affect performance or reliability for storage. I'd recommend using ZFS on Linux (Note, I still use FreeBSD as my primary OS for my personal laptop).
It's running in a 2U case I got from servercase UK that takes 6 hard drives, it's running:
- Core i3 9100T (35w TDP, configurable down to 25W)
- Asrock Rack WS246I (mini itx workstation board, no need for an HBA as there are 8 SATA ports on board, 4 standard and another 4 from the OCuLink)
- 32GB ECC DDR4 (2 16 GB sticks)
- Solarflare 7 series 10Gb SFP+ NIC (second hand, from ebay)
- 6 ironwolf 4TB NAS drives
Total cost was just a shade under 1000 GBP and it's racked up with my networking gear in the garage.
I don't know and don't care about TerraMaster's software (it might be awesome - I have no idea). I just rolled my own NixOS install with ZFS so that I could have a deterministic installation (I've heard good things about the TrueNAS OS as well, but I'm a control freak and like being able to rebuild the entire server with a single command and a config file, so I stick with NixOS).
The nice thing is that I essentially got a motherboard, CPU, PSU, and compact case for $350 (for the F2-422). All I had to do was upgrade the RAM (SO-DIMM) and add the drives.
I've long since reduced to only two drives for my NAS. At one point I was up to 7 drives before I realized my madness. It's cheap enough to get the storage I need with two mirrored drives, is quieter and uses less energy (I can keep it in the same room), and when I finally outgrow them in 5 years or so, the old drives will be re-purposed as backup via an external USB enclosure I keep around.
It’s unclear if one could install TrueNAS on a Synology, QNAP, Teramaster etc. Sometimes hardware is not supported.
Edit: Looks like someone did a writeup for TrueNAS on a TerraMaster: https://joelduncan.io/freenas-on-terramaster-f2-221/
Also: https://mightygadget.co.uk/how-to-upgrade-the-terramaster-f4...
You get an affordable TrueNAS server.
>Before building this system, I had zero experience with ZFS, so I was excited to try it out.
Sorry, but this is amusing to me. ZFS on TrueNAS is probably fine, but you're building your production NAS, to replace the Synology device you've become "so dependent on". Don't become dependent on ZFS without knowing the implications!
I was facing this choice recently, and I agreed with the other tech savy person in the household that we should just use good old LVM + Btrfs. Not only does it run like a charm, but it also allowed us to switch the LV from single (during the data move) to RAID 1 and eventually to RAID 5/6 with zero issues. It will also be much easier to recover from than ZFS.
On another note, it's a bad market to buy NAS drives, especially from Seagate. Seagate Exos drives are at this point in time often cheaper than IronWolf, even non Pro IronWolf. They're slightly more noisy and don't come with the free data recovery, but otherwise they're a straight upgrade over the IronWolf drives.
When it fills up, I delete some files rather than adding disks.
https://man7.org/linux/man-pages/man7/lvmraid.7.html#DATA_IN...
Benchmarks are nice, but you're comparing an old NAS to a brand new one with significantly more computer power, and had you compared to a new Synology, you'd probably arrive at the same findings.
Parts of you power consumption could very likely be from your choice of hard drives. The 8TB Seagate Ironwolf has an average power consumption of 8.8W [1], where a WD Red 4TB requires at most 4.8W [2]. With 4 drives, that's an additional 16W of power consumption right there.
Another thing you could try is to enable powerd and measure if it has any effect on the power consumption. In case it does, you can enable it in TrueNAS Core by adding it as a tunable via the web UI. System -> Tunables -> Add Type = rc.conf Variable = powerd_enable Value = yes
FreeBSD Wiki also has some Tuning Parameters [3]
[1]: https://www.seagate.com/www-content/datasheets/pdfs/ironwolf...
[2]: https://documents.westerndigital.com/content/dam/doc-library...
I ultimately decided against ECC RAM
Also unsurprised to see load-related issues stemming from his embedded Realtec hardware.
I purchased a dual xeon (24 cores total) with 64gb of memory, 12 3.5" bays, dual power supplies, for about $250 from a liquidator.
Filling it with HDDs was pricey but you can expand as you need it to spread the expenditure.
Results: - idle - 55W - full usage - 200W
Not bad for 10 years old server.
I am a newbie with server builds. Can you share an Ebay or Amazon link that can be a good/cheap starting point? Also, will these servers contain everything I need to get up and running (with the exception of hard drives, of course)?
Just check how many RAM they have, how many disk slots and what is their connector (SATA/SAS).
With drive bays you'll see: SFF (small form factors, ie 2.5" drives) and LFF (3.5")
Usually written like "12 LFF" for say a server that can hold 12 3.5" drives.
Look out for if it comes with drive trays for the bays or not. They're usually surprisingly cheap because there's a bazillion of them out there but it could be $5-$20 each.
Then there's SAS / Raid controller stuff you need to look out for in regards to compatibility with your configuration.
For example if you're going to run TrueNAS (or whatever it's called nowadays) you'll want to search "XXXXX controller TrueNas compatibility" to stumble upon a thread or some hardware list to hear what people have to say. There actually aren't too many models, or they're clones of each other with a Dell or HP branding label on it, so you're bound to find some stuff, for mainstream stuff sold on ebay anyway.
The hardware will typically be at least power-on tested, which for this kind of equipment is usually good enough, but beyond that your experience will vary.
There will be typical gremlin stuff like an unseated cable which will throw errors and have you wondering if it's a bad drive, or bad install, or whatever... but I think if you're building your own and bringing older gear back to life, you won't be able to avoid that unless you're buying a pre-packaged NAS box.
Another thing I'd like your opinion on is this: my use cases involve running a few services at home (NAS, Nextcloud, Jellyfin, Photoprism etc.) and store "family" data (documents, pictures etc.) but nothing "data hoarder" level. So, 4 TB would easily last us a decade or more.
Do you think such enterprises servers make sense for me? (Noise, complexity etc.) Or should I just get a used desktop/laptop?
You could get away with whatever machine you have lying around with enough space that does differential back ups to an external (or with something hosted somewhere else over the internet).
So, my options are to buy old desktop, old enterprise server or build a new desktop. The first two are similar price points on eBay, but I am not sure about the noise and power tradeoffs between the two.
Like, if a 1U server is to noisy and power-consuming compared to the desktop, probably not a good choice for my case.
This is not true at all. You can’t add new drives to an existing vdev, but you are free to add new vdevs to an existing pool whenever you want.
I built my own nas a few years back with a multi disk zfs pool and ubuntu running mellanox 10g nics. Fast as it was, the complexity of it all, especially zfs having its own way of doing everything, made it a time hog. Just wasn't worth it.
Now I run my nas in a Debian Mint VM with consumer grade 2.5G nic adapters. Simplicity and ease of recovery are my priorities.
[1]: https://openzfs.github.io/openzfs-docs/man/8/zpool-add.8.htm...
[2]: https://docs.oracle.com/cd/E53394_01/html/E54801/gayrd.html
True RAIDz expansion is something that's supposed to be coming, possibly in Q3 2022, so it may be that by the time one needs to expand a volume, that ability will have landed. That'll be a game changer.
I had all kinds of thermal problems with a too small case that I used for my truenas build. It would turn off without any trace in server logs (I have real server HW and therefore expected something in logs since there is a whole separate computer for this).
I changed the case from a NAS case to another fractal desfrign case with lots of space for drives and heatsink. All thermal issues disappeared.
I just wanted to warn anyone who is building to take this seriously. Some HW drives generate a lot of heat.
Having scheduled checks is a good idea: I have weekly short SMART tests, monthly long SMART tests, and quarterly data scrubs.
The TinyPilot device looks nifty - it's a Raspberry Pi as a remote KVM switch. I stumbled on that last night as I was banging my head against a familial tech support issue.
Also, in the bay area I like the local vendor https://unixsurplus.com/
I was obsessed with making an instant failover cluster. I never managed to get it working exactly how i wanted and it relied on two old UPSs with dead batteries to operate as STONITH devices (they had well supported rs232 interfaces).
I sometimes think about investigating that idea again but maybe with raspberry pis and cheap iot plugs.
I'm not sure if TruNas supports this kind of config, or if they enforce booting from a separate drive. But FreeBSD itself is perfectly happy to boot from ZFS.
My question: How do you safely store your physical backup drives/devices?
I have a fireproof box, but I don't think it was made for safely storing electronics in the event of a fire.
To be clear I meant the drives backing up the NAS, not the actual NAS.
I think backing up online ultimately is the safest choice, and it takes getting comfortable doing that and being okay with paying a fee. This is for data that I can't lose, like family photos, etc.
I started looking into using rclone directly from my FreeBSD NAS device. rclone seems to support many providers.
Did the price comparison for sonology a few years ago and felt it just made more sense to build my own. It’s just the current LTS Ubuntu release and it runs plex, pihole, file sharing, cups print server and some other stuff
On the other hand, I can’t stand people who say “homelab”. Ugh.
Is there a pointer to someone who does this but actually goes through the ECC grief?
It's really hard to chop through all the ECC "marketing" aka lies from the different motherboard manufacturers.
What's a cost effective CPU/mobo/ECC for NAS?
Is this how the math works? Does having more drives mean the individual drives themselves are more likely to fail? Is running 4 drives safer than 100?
Of course, we know that having a larger sample size and seeing more failures doesn't _actually_ mean that groups of disks are less reliable, but it could seem that way if you don't think too hard about it.
And to my account, I think my upsides are that:
- ability to choose the kernel
- no need for SSD for base OS since running off of RAM is rather easy on Linux
- samba can run in a container thus a bit more control security-wise
- server may run something else as well
Of course, this comes with a lot more technical hurdles. More like a side-project than utility really. That's why I was wondering does TrueNAS provide non-obvious upsides that would be lacking in self-rolled one.
The upsides are that it's plug-and-play for anyone who doesn't want to research all the options available and figure out the various pitfalls on their own.
> no need for SSD for base OS since running off of RAM is rather easy on Linux
I don't understand this sentence. You're running off a RAM disk with no boot drive? What if you have a power outage?
> samba can run in a container thus a bit more control security-wise
Core supports FreeBSD jails and Scale supports Docker so you could run samba in a container on either if you're willing to do set it up yourself.
> server may run something else as well
As before, both have jail/container functionality. I haven't used Scale myself but Core comes with a bunch of "click to install" jail options for stuff like Plex, ZoneMinder, etc. Our machine also runs a Windows VM (ew) and a Wordpress install in a Jail
> You're running off a RAM disk with no boot drive? What if you have a power outage?
Yes, the server only has the HDDs which contain the NAS data. The server bootloops until it gets an image from the router (ipxe boot). The disk images have systemd scripts which install everything from 0 on each boot. Coincidentally, this means system restart is how I upgrade my software.
> Core supports FreeBSD jails and Scale supports Docker
This clarifies the situation -- TrueNAS seems like an option that I would recommend for anyone who wants a quick OSS NAS setup.
I mean backing up the file system (as with ZFS send) not scanning all files (using rsync of restic).
ZFS send and receive is a good way to do it, but there is no ZFS send and btrfs receive!
https://forums.serverbuilds.net/t/official-recommended-sas2-...
There you go.
> 1200+ USD
I'm not sure what sort of timeline the author lives in, but that is not budget in mine.
it is ok for like an offsite backup that you'll touch maybe once in years, if it blows up one day, just upload a new backup
* https://www.techradar.com/news/larger-than-30tb-hard-drives-...
Buy multiple drives and a docking station and you can rotate them:
* https://www.startech.com/en-us/hdd/docking
ZFS send/recv allows for easy snapshotting and replication, even to the cloud:
* https://www.rsync.net/products/zfs.html
* https://arstechnica.com/information-technology/2015/12/rsync...
e.g. a 20TB drive from Seagate is $500. A 4TB drive is $70, 8TB is $140. Getting the same spend in smaller capacity drives would give you 28TB in the 4TB drives and 24TB/32TB in the 8TB drives (for $80 under/$60 over).
Add in a second to rotate and you're spending $1000 in drives, assuming these 26TB drives replace the 20TB drives at a similar price when they trickle down to consumer hands.
We found ZFS led to maintenance issues, but it was probably unrelated to the filesystem per say. i.e. culling a rack storage node is easier than fiddling with degraded raids.
For critical data though I use Borg and a Hetzner StorageBox.
I don't back up my Blu-Rays or DVDs, so I'm backing up <1 TB of data. The current backups are the original discs themselves, which I keep, but at this point, it would be hundreds of hours of work to re-rip them and thousands of hours of processing time to re-encode them, so I've been considering ways to back them up affordably. It's 11 TiB of data, so it's not easy to find a good host for it.
my budget homelab is 100% recycled (ewaste):
- dual core pc (free)
- hard drives (free)
- ram (free)
- lcd monitor (free)
- mdadm + ext4
At the beginning of the year 0.35€/kWh was a good estimate. An extra 10W for one year then costs about 30€.
I get the "recycled" motivation, but at that point you might be wasting lots of electricity (and as a result also money).
The number 1 trap you fall in to is during rebuild after a failed drive. In order to rebuild every byte on every other drive has to be read. On massive arrays this process invariably throws up additional errors, however this time you might not have the parity data to recover it. This process continues in a snowballing situation. This problem is exacerbated by using unsuitable drives. This author seems to have chosen well, but many choose to select drives for capacity over reliability in a quest for the most TB usable possible. A few years ago there was also the scandal of the WD Red drives that were totally unsuitable for RAID usage.
And to make matters worse there is the performance impact. Writing consists of 4 operations: read, read parity, write, write partity. That gives a /4 penalty on the sum of your arrays drives IOPS.
RAID6/Z2 gives you slight relief from the above risk, however at the increased cost of an additional performance hit (a /6 penalty)
If going RAID(Z), it is generally considered best practice to go for a model that includes a mirror. There are decisions to be made whether you stripe mirrors or mirror a stripe. Personally my preference for reducing complexity and improving quick rebuild is to stripe across mirrors. So that is RAID10. You pair your drives up in mirrors, and then you stripe across those pairs. The capacity penalty is 50%. The performance penalty is close to zero.
The author also chose to skip a write buffer (ZIL) drive. This, imo, is a mistake. They are a trivial cost to add (you only require a capacity that gives you the maximum amount of data you can write to your array in 15 seconds (tunable)) and they offer a tremendous advantage. As well as gaining the benefit of SSD IOPS for your writes you also save wear on your data array by coalescing writes in to a larger chunk and buy yourself some security against power cuts etc as faster IOPS give you a reduced likelihood of coinciding with an environmental issue. And if you are especially worried you can add them as a mirrored pair.
You can also add SSDs as a cache (L2ARC) drive (I think the author missed this in their article) to speed up reads. In the case of the authors use case this would really help with things like media catalogs etc as well as buffering ahead when streaming media. The ARC in ZFS always happens, and the L1 is in RAM, but a L2ARC is very beneficial.
The author did comment on RAM for the ARC and sizing this. ZFS will basically use whatever you give it in this regard. The really heavy use case is if you turn on deduplication but that is an expensive and often unnecessary feature. (An example good use case is a VDI server)
Last tip for ZFS: turn on compression. On a modern CPU it's practically free.
Either I misunderstood or there are some typos but this math seems all kind of wrong.
A 4% risk per year (assuming failure risk is independent of disk age) is less than 0.1% by week. A 2% risk per week would be a 65% risk per year!!
2 simultaneous failures at the same week for just 2 disks (again with the huge assumption of age-independent risk) would in the order of magnitude of less than 1:10^6 , so more than 20k years(31.2 k years tbc)
Of course you either change your drives every few years so the age-independent AFR still holds or you have to model the probability of failure using some exponential distribution like Poisson's. Exercise for the reader to estimate the numbers in that case.
I've run ZFS for home storage and work backups for ~15 years, across Nexenta, ZFS-fuse, FreeBSD, and OpenZFS, backing up hundreds of machines, and have never lost data on one of them.
It was also very complex to manage compare to ZFS, with many different layers to consider.
I'm sure it shines in a data center, for which it has been designed. But unless something radical has changed in the last year, it's not for a budget homelab NAS.
Seen too many people's TrueNAS/FreeNAS installs glitch up over the years to trust the zfs community edition as a sane production choice. ZFS certainly has improved, but Oracle is not generally known for their goodwill toward the opensource community. ;-)
Overall been very little fuzz for my home NAS system.
BTRFS seems to be maturing nicely, hopefully we can start using it for these types of workloads in the next few years.