RAM?
Everytime I looked into setting up a freenas box, every hardware guide insisted that ungodly amounts of absolutely-has-to-be-ECC RAM was essential, and I just gave up at that point.
Everytime I looked into setting up a freenas box, every hardware guide insisted that ungodly amounts of absolutely-has-to-be-ECC RAM was essential, and I just gave up at that point.
I don't see why ZFS would be more prone to data integrity issues spawning from a lack of ECC than any other filesystem.
If I were running a server farm or something, then yeah, I'd probably use ECC memory, but I think if you're running a home server, then the argument that ZFS necessitates ECC more than Ext4 or Btrfs or XFS or whatever doesn't really seem to be accurate.
Agreed.
> If I were running a server farm or something, then yeah, I'd probably use ECC memory, but I think if you're running a home server
Then you should still use ECC RAM, regardless of what filesystem you're using.
No, really. ECC matters (https://news.ycombinator.com/item?id=25622322) generally.
https://www.truenas.com/community/threads/ecc-vs-non-ecc-ram...
(the gist of the scary story is that faulty ram while scrubbing might kill "everything".) However, in the end ECC appears to NOT be so important, e.g., see
FreeBSD Mastery: ZFS by Michael Lucas around pg 174
Deduplication Memory Needs ==========================
"For a rough-and-dirty approximation, you can assume that 1 TB of deduplicated data uses about 5 GB of RAM. You can more closely approximate memory needs for your particular data by looking at your data pool and doing some math. We recommend always doing the math and computing how much RAM your data needs, then using the most pessimistic result. If the math gives you a number above 5 GB, use your math. If not, assume 5 GB per terabyte."
https://www.tiltedwindmillpress.com/?product=fmzfs
This is not to say you need 5GB for every 1TB of data. It doesn't even mean you need 5GB of data for every 1TB for which you have enabled dedup it means you need approximately 5GB of data for each TB of data which is both duplicated and residing on a dataset for which you have enabled dedup. Because of the high memory cost of dedup which rises exactly in proportion to its utility its only useful in cases in which you can plan ahead for its requirements. 99% of users are unlikely to use dedup however this doesn't stop some, not you obvious, from promoting the idea that ZFS requires 5GB of memory per TB or some some absurd figure.
As an aside I really liked the book I found it easy to read and understand and very informative despite being focused on FreeBSD its mostly applicable to Linux as well.
ZFS defaults to assuming it is the primary reason for your box to exist, but it only takes two lines to define more reasonable RAM usage: zfs_arc_min and zfs_arc_max. On a NAS type server, I would think setting the max to half of your RAM is reasonable. Maybe 3/4 if you never do anything except storage.
ECC is not recommended because ZFS has some kind of special vulnerability without it; ECC is recommended because ZFS has taken care of all the more likely chances of undetectable corruption, so that's the next step.
For a long running, non-idle system, a good rule of thumb is that all RAM not being actively used is being used by evictable caching.
total used free shared buffers cached
Mem: 12286456 11715372 571084 0 81912 6545228
-/+ buffers/cache: 5088232 7198224
Swap: 24571408 54528 24516880ECC tends to attract zealots after a perfect error-free existence which ECC does tend towards but doesn't deliver, it just reduces errors. I personally don't care about a tiny amount of bit rot (zfs will prevent most of this) and rebooting my storage machine now and then.
You can run ZFS/freenas on a crappy old machine and you'll be just fine as long as you aren't hosting storage for dozens of people and you aren't a digital archivist trying to keep everything for centuries.
Real advice:
* Mirrored vdevs perform way better than raidz, I don't think the storage gain is worth it until you have dozens of drives
* Dedup isn't worth it
* Enable lz4 compression everywhere
* Have a hot spare
* You can increase performance by adding a vdev set and by adding RAM
* Use drives with the same capacity
To add to that, ZFS dedup is a lie and you should forget its existence unless you have a very specific scenario of being a SAN with a massive amount of RAM, and even then, you had better be damn sure.
I really wish ZFS had either an option to store the Dedup Table on a NVMe like Optane, or to do an offline deduplication job.
Now, that becomes the only place entries on it are stored, so you best make it redundant if you don't want to lose your pool from a single NVMe failing, but the feature is there.
The latter I would predict seeing approximately when the sun burns out, on ZFS. It _really_ doesn't like the idea of data changing locations retroactively.
I'm going to have to do some test setups with this.
Is the perf penalty low enough now that it just doesn't matter? I've always disabled compression on datasets I know are going to store only high-entropy data, like encoded video, that has a poor compression ratio.
I second the hot spare recommendation many times over. It can save your bacon.
usually it will hit nothing and have no side effects
Sources: - https://www.reddit.com/r/DataHoarder/comments/3s7vrd/so_you_... - https://www.reddit.com/r/homelab/comments/8s6r2r/what_exactl... - My own tests with around 8 TB ZFS data in a Linux vm with 256 MB RAM.
I have several file-servers all use ZFS exclusively. and 10x that number of servers using ZFS as the system FS.
Rule of thumb that I like: 1GB RAM/TB of storage. This seems to give me the best bang-for-our-buck.
For a small (under 20) number of office users, doing general 'office' stuff, using Samba, it's overkill.
For large media shares with heavy editor access, and heavy strains on the network, it's a minimum.
Depends on what the server is serving.
DeDUP is a different story. The RAM is used to store the frequently accessed data. If you are using DeDUP you fill the motherboard with as much RAM as will fit. NO EXCEPTIONS! This may have been the line of thinking that scared you away from it.
I have a 100TB server that is just used for writing data to and is never read from (sequential file back-ups before it's moved to "long term storage"). It has 8GB of RAM, and is barely touched.
I also have a 20TB server with 2TB of RAM, that keeps the RAM maxed out with DeDUP usage.
ECC: It's insurance, and it's worth it.
You effectively (unless you use allocation classes) need to keep the entire DDT in RAM all the time if you don't want any write to a dedup-enabled dataset to potentially require blocking on reading the relevant segment from spinning disks into RAM (thus tanking performance even worse than dedup normally does). It's not really related to the mechanisms in the rest of ZFS for keeping {frequently,recently} used data cached in RAM.
https://www.freenas.org/hardware-requirements/
I myself use freenas with 16GB of non-ECC ram.
Of course it is possible to have a bit flip in memory that is then dutifully stored incorrectly by ZFS to disk, but this was a possibility without ZFS as well.
I've actually been waiting for this feature for since I first setup my pool. It seemed theoretically possible we were just waiting for an implementation.
ZFS only needs a lot of RAM if deduplication is enabled. And it shouldn't be for most use cases, or only enabled on one dataset that benefits from it.
Many ZFS installs are fine on 8GB or less.
ECC RAM is better but not required. The idea is to catch memory errors, hence ECC is better.