Hacker News Reader

Comment by speff | Hacker News Reader

Q - assuming the NAS was strictly used as NAS and not as a server with VMs, is there a point in having a large amount of RAM? (large as in >8GB)

I'm not sure what the benefit would be since all it's doing is moving information from the drives over to the network.

View on HN

I am not at all an expert, I can only share my anecdotal unscientific observations!

I'm running a TrueNAS box with 3x cheap shucked Seagate drives.*

The TrueNAS box has 48GB RAM, is using ZFS and is sharing the drives as a Time Machine destination to a couple of Macs in my office.

I can un-confidently say that it feels like the fastest TM device I've ever used!

TrueNAS with ZFS feels faster than Open Media Vault(OMV) did on the same hardware.

I originally setup OMV on this old gaming PC, as OMV is easy. OMV was reliable, but felt slow compared to how I remembered TrueNAS and ZFS feeling the last time I setup a NAS.

So I scrubbed OMV and installed TrueNAS, and purely based on seat-of-pants metrics, ZFS felt faster.

And I can confirm that it soaks up most of the 48GB of RAM!

TrueNAS reports ZFS Cache currently at 36.4 GiB.

I dont know why or how it works, and it's only a Time Machine destination, but there we are those are my metrics and that's what I know LOL

* I don't recommend this. They seem unreliable and report errors all the time. But it's just what I had sitting around :-) I'd hoped by now to be able to afford to stick 3x 4TB/8TB SSDs of some sort in the case, but prices are tracking up on SSDs...

ZFS uses a large amount of ram, i think the old rule of thumb was 1GB ram per 1TB of storage

That's only for deduplication.

https://superuser.com/a/993019

I do like to deduplicate my BitTorrent downloads/seeding directory with my media directories so I can edit metadata to my heart's content while still seeding forever without having to incur 2x storage usage. I tune the `recordsize` to 1MiB so it has vastly fewer blocks to keep track of compared to the default 128K, at the cost of any modification wasting very slightly more space. Really not a big deal though when talking about multi-gibibyte media containers, multi-megapixel art embeds, etc.

Have you considered "reflinks"? Supported as of [OpenZFS 2.2](https://github.com/openzfs/zfs/pull/13392).

Haven't used them yet myself but seems like a nice use case for things like minor metadata changes to media files. The bulk of the file is shared and only the delta between the two are saved.

Neat; will look into this. My setup is several years older than this, predating even FreeBSD's move to OpenZFS, and I just haven't touched the config of it since then since it works flawlessly (and since I already bought the RAM lol)

cross-seed | cross-seed https://www.cross-seed.org/

I believe they are saying they literally edit the media files to add / change metadata. Cross-seeding is only possible if the files are kept the same.

ZFS also uses RAM for read through cache aka ARC. However, I’m not sure how noticeable the effect from increased RAM would be - I assume it mostly benefit for read patterns with high data reuse, which is not that common.

Huh. More than just the normal page cache on other filesystems?

Yes. Parent's comment matches everything I've heard. 32GB is a common recommendation for home lab setups. I run 32 in my TrueNAS builds (36TB and 60TB).

You can run it with much less. I don't recall the bare minimum but with a bit of tweaking 2GB should be plenty[1].

I recall reading some running it on a 512MB system, but that was a while ago so not sure if you can still go that low.

Performance can suffer though, for example low memory will limit the size of the transaction groups. So for decent performance you will want 8GB or more depending on workloads.

[1]: https://openzfs.github.io/openzfs-docs/Project%20and%20Commu...

ZFS will eat up as much RAM as you give it as it caches files in memory as accessed.

All filesystems do this (at least all modern ones, on linux)

It depends on your file workload. The RAM can be used as a read cache.

I have some workloads where I have to go through a lot of files multiple times and the extra RAM cache makes a huge difference. You can tell when the NAS is pulling from cache or when it has a cache miss.

As the other said already if you have more RAM you can have more cache.

Honestly it's not that needed but if you would really use the 10Gbit+ networking then 1 second is ~125Mbytes. So depending on your usage you can never even more than 15% utilization or have it almost all if you constantly running something on it ie torrents or using it a SAN/NAS for VM on some other machine.

But for a rare occasional home usage nor 32Gb nor this monstrosity and complexity doesn't make sense - just buy some 1-2 bay Synology and forget about it.

I won’t be able to sleep having my data just on 1 disk

No matter what you should have an off-site backup as well in case of lightning, flood, fire, virus, etc.

If you use ZFS you might need more RAM for performance?

People get carried away with their home lab setups. There's a distinct type of person that thinks they need 100tb of storage in their own house.

If you're running a NAS for a company that has many users and multi disc access at the same time, sure. But then you're probably then not buying hdds to shuck and cheap components off ebay.

My NAS:

1) A refurbished Dell Wyse 5070 (8GB of RAM) with a cheap 64GB SSD from 2013 2) An 8-bay USB-C hard drive enclosure 3) 4 used 12TB hard drives from eBay, 4 3TB drives from 2010 that still somehow haven’t died. 4) A headless Debian with various Linux ISO trackers / downloaders in Docker / Plex (the CPU, though slow, has decent hardware encoding) 5) No RAID, but an rsync script for important Linux ISOs and important data that runs weekly across different drives. I also have cold storage backup by purchasing “Lot of X number” 500GB hard drives on eBay from time to time which store things like photos, music, etc over two drives each.

The whole setup didn’t cost me much, and is more than enough for what I need it to do.

No Raid? Have you considered using SnapRAID (+ mergerfs if you fancy)? Seems like an ideal use case for the solution.

Caching files in ram means they can be moved to the network faster - right?

Depends on the network speed. At 1Gbps a single HDD can easily saturate the network with sequential reads. A pair of HDD could do the same at 2.5Gbps. At 10Gbps or more, you would definitely see the benefits of caching in memory.

Not as much as expected. I have several toy ZFS pools out of ancient 3tb wd reds, and anything remotely home-grade (stripped mirrors, 4,6,8 wide raidz1/2) saturates the disks before 10gig networking. As long as it's sequential, 8gb or 128gb doesn't matter.

Makes sense. I didn't know if the FS used RAM for this purpose without some specialized software. PikachuEXE and Mewse mentioned ZFS. Looks like it has native support for caching frequent reads [0]. Good to know

[0]: https://www.truenas.com/docs/references/l2arc/

ZFS cache.

Reply on news.ycombinator.com