Comment by zamadatix | Hacker News Reader

Comment by zamadatix | Hacker News Reader

I did something similar this year with 12x 4tb NVMe drives. I had a 12x 2tb setup is an fashstor 12 but the CPU and RAM limitations led me to traditional RAID there. For the second NAS I just went standard parts but with extreme performance, 13900K and the like. To get the necessary PCIe lanes (since not every port would bifurcate) I used 3 4 way switches from aliexpress. Choosing a motherboard with 2 x8 cpu slots made the bandwidth not a problem, the third does hang off the demo 4.0 lanes though. The boot drive hangs directly off the cpu in the x4 slot. The 13900K actually supports ECC in the workstation motherboards as well. I just expose everything via NFS and call it a day

All in all I'm happy with it. I still use spinning rust for the backup though, no sense worrying about that. At the time it was cheaper to get 12x4 TB budget NVMe and the extra parts than go for any reasonable count of SATA SSDs, not sure if that's still true or not. SATA definitely would have been easier, e.g. stuff like in the Flashstor 12x2TB build I had to submit a kernel patch because the cheap drives were duplicating their nsids.

> the CPU and RAM limitations led me to traditional RAID there

IIRC there was some talk that RAID-Z lead to write amplification compared to mirrored drives, thus not good for cheaper SSDs. Haven't had time to sit down and think it through, does anyone know if thats right or wrong?

> I used 3 4 way switches from aliexpress

Did you find any that were significantly cheaper than ~100 USD?

In general I've found cheap SSDs these days have better endurances compared to cheap spinning disks anyways. https://www.servethehome.com/discussing-low-wd-red-pro-nas-h.... I think the drives I got were 1.6 PBW rated, no errors or failures yet. Beyond that, my workload isn't write heavy enough for this to be a concern of mine whether or not there is amplification. That SSDs don't count reads or active drive time in the endurance rating was more important. For the Flashstor build I did RAID4 instead of RAID5 just because I could though.

I want to say that sounds about right on price. If I didn't also have a desire for high single core performance at the same time a used/old Threadripper/Epyc build and bifurcation would probably make more sense. I also disconnected the onboard tiny "definitely going to be noisy as hell in 3 months" low quality fans from them and just rested a 140mm blowing down across the top of the 3 cards at 30% speed. Temps of the controllers and SSDs became better and it's dead silent.

Can you expand more on the hardware used? Specifically, which 4 way switches you used?

I have a bunch of unused M.2 drives from decommissioned servers that I'd love to use as additional storage.

I started with this page for adapters people had tried and recommended here https://forums.servethehome.com/index.php?threads/multi-nvme... but I think I ultimately went with the LinkReal x8 to quad.

If you don't specifically want the high per core performance of something like a 13900k a used/old Threadripper or Epyc system and bifurcation might make more sense. It'll also enable you to get maximum per drive bandwidth, if that's a concern for your (when you have 12 drives in some form of stripe and parity the per drive bandwidth ends up not being that important for most sane workloads though).

What brand of 4tb nvmes are you using?

I scooped up some MP34s on sale. The have something like a 1.6 PBW endurance rating and I haven't had a single one fail or start throwing errors yet (though my workload isn't as write heavy as many might have).

That's cool. I use some 4 tb silicon power brand drives. I didn't research them much. Prices are definitely going to fall soon on 8 tb, and I'll likely retrofit things pretty quickly with those.

Reply on news.ycombinator.com