Build a Raspberry Pi CM4 4-Bay NAS with Wiretrustee Carrier Board | Hacker News Reader

Build a Raspberry Pi CM4 4-Bay NAS with Wiretrustee Carrier Board

cnx-software.com

115 pointstobijkl72 comments

There are a few people currently working on NAS boards based on the CM4. A few different approaches in terms of chipsets used and performance targets.

I'm trying to track them in this GitHub issue [1] but a couple people have remained more or less anonymous as they don't want to attract attention too early.

The single PCIe lane is the biggest limitation if you're looking for raw speed (350 MB/sec is kind of the upper real-world sustained transfer limit), though since the gigabit Ethernet port is on a different interface, you can still expect to get 80-100 MB/sec network transfer speeds.

Something like this, with the right case and OMV or other adequate software would be a relatively competitive replacement for low-end NASes.

I'm also exploring building a 2.5G NAS with a CM4, but the PCIe bus speed limitation is what kinda hamstrings that. Hopefully the next Pi revision has a 4x (at least) lane, like the RockPro64.

[1] https://github.com/geerlingguy/raspberry-pi-pcie-devices/iss...

Have you come across anyone producing a PoE board for the CM4 - just looking for a network port, USB and perhaps SD card (similar to a Ubquiti Cloud Key)?

(the Gumstix camera board is overkill for my needs)

Why not use the regular Raspberry Pi 4 with the PoE module?

The CM4 is only useful in most cases if you want to use the PCIe bus for something else, or if you’re trying to embed it in another product.

The existing PoE module is noisy as hell, and due to it's design it's a PITA to fit a different fan

CM4 on a PoE baseboard offers far more cooling options

Waveshare has one. Haven't used it personally but it seems to suit your needs. Though at ~$50 you're probably better off just getting a Pi 4B + PoE hat.

https://www.waveshare.com/compute-module-4-poe-board.htm

Can POE provide enough power to turn SATA drives? I kind of figured for a NAS it probably makes sense to go with a regular power supply, especially since it’ll be mounted in an easy to access location for drive replacement.

802.3at specifies max 30W at the output port at 600mA so in theory you could shunt that to 5V and get about 5A after losses. Enough to spin up a 3.5" SATA.

But most consumer retail PoE switches are only .3af compliant which give 15W minus transmission losses.

802.3bt can provide higher power, easily enough to run a few HDDs but there are few switches on the market that offer 802.3bt. Almost nothing available for the residential/consumer space.

A regular Raspberry Pi with a cheap POE USB-C splitter from Aliexpress works very well.

https://a.aliexpress.com/_mMpvxGV

"Have you come across anyone producing a PoE board for the CM4 ..."

Wait, I thought the official CM4 breakout/dev board (the one with the PCIe slot on it) had PoE, right ?

80MB/sec is actually quite fine for most NAS applications.

Anyone trying to get 10G speeds out of a high end NAS won’t be looking to Raspberry Pi solutions right now, anyway.

The real advantage, IMO, is that this helps kick off the popularity of DIY NAS solutions based on ARM hardware. It’s not the first solution in this space, but Raspberry Pi is great for taking things mainstream.

I imagine that a few years from now we’ll have an even faster Raspberry Pi to build a NAS around. These current-gen solutions might be just what we need to get the software sorted out before the powerful hardware arrives.

Its sorta hard to justify those at the moment, since the XHCI interface on the base Pi4B is more than capable of maximizing the PCIe x1 interface. Basically, you can run a NAS at the capabilities of the machine with random USB3 JBODs.

So a $100 5+ bay USB3 JBOD, RPi4B run over the 1G nic, and a 64-bit arm64 distro (because rasbian will die with modern SATA disks in a larger capacity JBOD) and you have a fairly reasonable low end NAS for basically the price of the disks.

(or just plug in 4 of USB3 easystores/etc and save on the enclosure).

I’ve done this (JBOD with USB 3.0 drives on a Pi 4) and it’s surprisingly effective.

I can easily saturate the 1G NIC which is more than sufficient for my use case.

I don’t use RAID or ZFS so neither the CPU or RAM are limiting factors.

"There are a few people currently working on NAS boards based on the CM4."

I continue to appreciate the time and attention you are paying to these new boards on your blog and in your comments here at HN - thank you.

I am having trouble sourcing maxed out CM4 parts - that is, 8 GB ram and 32 GB onboard storage[1]. They are either sold out until August or October or something silly like that or they are only available in 200+ quantity.

Do you have any suggestions as to where I could source those ?

[1] CM4008032, I think ...

Nothing against cool RPi projects but be aware that a motherboard with a NAS-grade Celeron, like ASRock J4105M, sells for $85 or so (but needs RAM, $20 for 4GB roughly). At $105, this leaves this carrier board around $35 of budget if you can get RPi CM4 4GB for $70. Plus all the doubts about reliability mentioned above.

Presumably that board will need a separate PSU, RAM, storage for a boot drive, etc?

Also it only has two SATA ports so would need a PCIe expansion card to match the four ports offered by the Pi board?

I suspect the Pi is also significantly more power efficient?

There are similar motherboards with 4 SATA ports. I have an older Gigabyte GA-J3455N-D3H with 4 SATA ports and two gigabit network controllers that cost in the same ballpark as GP's model.

RAM is accounted for in GP's description.

Regarding the boot drive, you can either boot from the data disks (say via a common ZFS pool or a common raid + lvm) or if you're happy with a SD card for the PI, I suppose a small USB drive would do the trick.

Regarding the PSU, but you'd need a beefy adapter anyway for running 4 drives, which isn't cheap, and there are cheap pico-atx PSUs available. I bought a compact case + PSU for my board for 50 euros around two years ago [1] that's still going strong (although I'm only running a single SSD in it), so the PSU alone should be less than that.

There's of course the power efficiency question and I presume this setup would be somewhat more power hungry than an RPI. The Intel specs for the J3455 give it at 10W TDP. I have no idea how much RAM consumes, but it's likely a few more watts.

[0] https://www.gigabyte.com/Motherboard/GA-J3455N-D3H-rev-10#ov

[1] https://www.inter-tech.de/en/products/case/mini-itx-nuc/itx-...

> Regarding the PSU, but you'd need a beefy adapter anyway for running 4 drives

How about 2.5 inch drives? I'm thinking about: https://cdn-reichelt.de/bilder/web/xxl_ws/A300/RPI_NAS_4XSAT...

I went looking for 2.5 inch NAS/ video recorder drives this past weekend and came up empty handed. I’d hoped to install them into a mini-pc NVR. The form-factor doesn’t appear to lend itself to the reliability drive manufacturers target for this class of drive. They’re optimized for high write speeds and very high write loads. You can of course use consumer laptop drives in a NAS, but write speeds will be lower and reliability may suffer.

> speeds will be lower and reliability may suffer

1Gbps is the max. anyways for my home setup, no matter how fast the drive is and I have a backup for important stuff, I'm not dependent on enterprise NAS drives.

I want: extremely small, low cost and very low watt usage.

The main issue is that larger capacity drives (> 2TB) seem to be SMR in the 2.5" range. I've looked and haven't found a PMR one (except maybe for "enterprise" drives which cost more than an SSD). Depending on how you want to use those drives, you may be having a bad time. For example when rebuilding a ZFS pool. [0]

If your use case / setup allow you to deal with the complete loss of your array (which, granted, is not 100% likely) or don't use ZFS, then I suppose you could look at Seagate's 2.5 Barracuda line. They're relatively cheap given their capacity and I don't think they're particularly unreliable in and of themselves.

[0] There are many people talking on the internet about ZFS performance with SMR disks. Here's a quick find:

https://arstechnica.com/gadgets/2020/06/western-digitals-smr...

RAID-Z2 on 4x2TB 2.5 inch HDD (without SMR) on a PI over 1 GbE sounds fine to me.

It's not super much space, but any two drives can fail, it's rather cheap and low powered + passive cooling should be possible.

Regarding power usage, I'm not very familiar with how those drives compare [0]. To be honest, power consumption was of interest for the build I was talking about because I wanted it to be fanless (I actually ended up adding a very silent fan, just for peace of mind). For my NAS needs, I'm building in a recycled server, so power usage isn't as much of a concern.

However, I did look for 2.5" drives (the server comes with a 2.5" backplane) and there don't seem to be many "consumer" drives in this format that are both high capacity (>2 TB) and non SMR (I'll be running ZFS). I'm also not looking to spend very much, so SSDs are out.

[0] Seagate specs for 2.5" Barracuda: https://www.seagate.com/www-content/datasheets/pdfs/barracud...

The startup current is 1.2 A under 5V. So if you're running four of them, you'll need 24 W just for the drives. Not sure if the "cheap" enclosures are able to stagger drive startup.

My Gigabyte motherboard cannot, so it would require a PSU large enough to drive both the board (CPU + RAM + etc) and the drives for a little while. So with four drives, you'd be looking at 50 W to be sure it fits.

50W is way more than I am willing to operate. I want it to be extremely low powered. I think that a 2.5 inch SSD might be a better fit for me then, maybe M.2 if there's a adapter for the PI.

SSD and low cost is still possible, but one has to make compromises in terms of storage size. A 1 TB network drive would be enough for my use case, be it 4 x 256 SSD's or one big.

Peak power is significantly different from operational power on many of these setups. The intel setups, frequently idle at a 1/10th or less the rated power. The rpi4 OTOH, isn't nearly as dynamic, particularly if your running 64-bit debian. Its still just a few W, but its not going to drop down into the mW range when its idle because debian/etc doesn't have particularly good power mgmt on that board.

So in the end the rated W can be quite misleading, particularly for home use where the NAS will tend to sit idle a large part of the day.

Okay, but what I really wanted to say is that I want a very small and low powered NAS without a big PSU. The PI is powered over USB, that's fine for me.

Well there are mITX boards with 8 or more SATA ports, and a lot of mATX boards with the same.

But the key differentiator IMHO is ECC and 10G. If one picks a board like https://www.gigabyte.com/Motherboard/C246N-WU2-rev-10 it then becomes possible to put a i3-9100/etc on it and gain ECC support. Given the 8 onboard SATA ports, and a 10G+ nic in the PCIe slot, you have a NAS that can best nearly everything on the market below a couple thousand dollars. There of course is a pretty wide range of atom/xeon boards with BMC's, and all of the above including the 10G. In the past I ran a https://www.asrockrack.com/general/productdetail.asp?Model=C... which has 12 sata ports, ECC and a BMC.

But, yes it will consume more than a rpi, and CPU power rating isnt everything. Having 8+ sata ports, bmcs, 10G ethernet, etc all up the board power consumption. But on the other hand, that i3 is rated for 65W, but idles at ~5 on a modern board, which is in the ballpark of the 2-3W of my RPi4 (which only swings a watt or so, and doesn't include the USB3 hub+devices).

So, given the NAS spends a lot of its time idle, and the power only spikes (and never to anything close to 65W) when its being used, it seems the extra power is worth the extra performance. I could probably fix the max frequency at 2Ghz like the RPi and save even more power.

I recently picked up two 36 bay sata super micro 4Us on ebay for ~$600 each.

The real cost is drives, not the thing you plug them into.

I've noticed this regularly with RPi projects. They are interesting, but the costs are often at least as high as an, often more powerful, x86 equivalent.

Keep in mind that the original goal of the Raspberry Pi foundation was directed toward education.

Raspberry Pi has inspired a lot of people to get their hands dirty with Linux and embedded systems, even if it’s not the optimal device from a pure engineering perspective.

In that regard, I’d call the Raspberry Pi a resounding success.

This NAS board is only for the Pi CM4 "Compute Module" - that is aimed at enterprise/embedded solutions, not education like the standard Pi boards.

I think for this, it's absolutely appropriate to compare to other solutions on market. Even the official Pi foundation data sheet describes CM4 as for "deeply embedded applications".

https://datasheets.raspberrypi.org/cm4/cm4-datasheet.pdf

I would also recommend to carefully examine low-cost x86 options, they often do come out ahead given they usually include all the stuff the Pi Foundation don't supply you - a boot volume, case, power supply etc.

Yes, I know, and I've used it in a project already. The CM4 still operates within the same Raspberry Pi ecosystem. Anyone familiar with the Raspberry Pi software will feel right at home on a CM4.

I agree, and I think they are phenomenal at that. There isn't anything else that combines the versatility, stability, and cost quite so elegantly for single Pi use cases for education or light desktop use. I use several VLC clients feeding monitors, for example, and they are fantastic.

But I also see things like 10 node clusters with a total of 40GB of RAM, a bunch of slow cores, and materials costs of $500+. That's harder for me to understand, outside of the "looks cool" metric.

I was thinking the same thing.

I love the pi and have many of them. But their beauty seems to come out at the edge where things are low power and they replace nonsense IOT devices.

To replace a server or desktop is a stretch. That's where using a pi is death by 10,000 papercuts. The super-competitive low end PC market gives you so many things for free, like good power supplies, a wide variety of cases and silly things like power switches and a clock with a battery.

Last year I built a nas system around the Asrock j4105 itx board, using a flashed dell perc h200 as an hba, in a fractal node 304.

I had looked at arm options, including the kobol 64 and boards from hardkernel, but decided that paying a bit more for x86 would make things a lot easier.

It's working well so far: low power draw and near silent, allows for proper zfs on linux support, hardware accelerating transcoding for Jellyfin streams, and at a bit over half the cost of a 6 bay x86 Synology.

Do not buy ROCK Pi SATA HAT. I own one, and after few months of lite usage with 4 SSDs, one channel (two SSD connectors) has just died. I just installed OpenMediaVault and configured LVM on 2TB, 2x 1TB and .5 TB SSDs, to use it as a home media server.

I recently tried to initialize warranty process (I bought it directly from shop.allnetchina.cn with shipping to Central Europe) but I got no replay.

What warranty process? You are in Central Europe. Seller is in China, no distributor in between. Basically you have no rights, buy it cheap and forget. Warranty lol. Best case for you is to open a case with your credit card vendor or PayPal. But you got what you paid for. As I maker in Europe I don’t feel bad about you, because I must sell everything for insane price with included 2 years warranty. While Chinese not.

That looks a lot like the mainboard from the Western Digital Sharespace I had about 10 years ago.

I was recently wanting a NAS again, but came to the conclusion that the best bang for your buck was just grabbing a cheap mini-ITX board and processor as the price of dedicated NAS boxes was not much different and was far less flexible. Then you would have full-fat sata, even m2 for a boot drive, whatever RAM you wanted etc.

I know it's not a very exciting solution...

Title is misleading?

End of the article : There’s close to no information about the software right now, and the hardware is not available yet

Nice, but the Gen10 HP ProLiant Microserver is my NAS platform of choice.

It supports ECC and has 4 cable-free swappable bays plus space for an SSD system drive in the top. The bolts-as-caddy system is also a great idea.

My memory is hazy now but there was something weird/screwy about the previous gen Microserver ... like, you couldn't use all four SATA drives at the same time as booting from the onboard ? Or something ?

How is the gen10 ? Can I use all four SATA drives for my raid array but still have a SSD/m.2 boot drive ?

I actually never took mine apart to put the SSD in the top slot, I only know it’s possible. (There is an additional mounting kit to support it on the Gen10.)

For me, three mirrored ZFS 4TB drives are all I needed for my local storage to be available and resilient. The fourth SATA slot is the SSD in a cradle.

My long term bet for safety is offline USB mirrors on-site, Borg off-site to rsync.net (thanks!) and tarsnap off-site for the smaller stuff.

Maybe one day I’ll be brave and use ZFS to mirror the content to rsync.net? I haven’t upgraded beyond ZoL 0.7 yet but if I did can I replicate an encrypted pool to you without giving you the keys?

You can have a lot of fun with Pi's, but I'm not sure I want to build a multi-TB NAS using a Raspberry Pi with a pretty rare addon board..

If you do want something like this, I think the ODROID-HC4[1] is probably a better option.

1 - https://ameridroid.com/products/odroid-hc4

Or just use USB3 - SATA connectors...

Yeah, USB drives with their own power supplies work very well with the Pi, but I can imagine high workloads could quickly run into problems and/or bottlenecks.

True, though that bottleneck would mostly be with the Pi itself. After all it only has GbE built-in, a single HD will easily saturate that for sequential loads or all loads for a SSD.

I have 8 disks hooked up to a Pi4 through USB3-SATA[1] with external 12V, works quite well in terms of reliability (had to turn on quirks mode[2] though). As I mentioned the main limitation is the single 1GbE interface, but this is just for archive data so.

[1]: https://www.aliexpress.com/item/4000001204302.html

[2]: https://www.raspberrypi.org/forums/viewtopic.php?t=245931

Yep, I've also noticed some unreliability when doing things like adding a mirror drive; at some point the USB controller just seems to fall off the horse and everything crashes.

My setup mainly streams video/music over wifi and it's rock solid at lower speeds. I wouldn't rely on it for any heavy duty storage needs though.

2 disks is pretty limited, is there a 4+ disk version of this?

I would still defer to something with ECC RAM and ZFS for keeping my data, but I could see myself using something like this for unimportant data and/or serving important data as a read-only mirror. Either way, I love that an RPi system will soon come in a sweet form-factor optimized for NAS use.

As a cheap offsite destination for backup,(assuming you have multiple destinations) this looks quite compelling.

Although I don't think ZFS is stable on the pi just yet[1], so perhaps 4 sata drives is overkill.

I would be interested in a single sata/cm4 carrier board though.

[1]citation needed, I've not really looked...

I wouldnt trust my data with this kind of devices. There's nothing wrong per se, but I have my doubts about how easy you can find replacement parts if something breaks in, say, a year or two.

If you really depend on your data and want it to be safe, i'd recommend spending the extra money and either getting a proper nas (synology/qnap) or go the proper diy way (aka an x86 box and truenas/unraid).

you really don't want to be in the position where you absolutely need your data but the replacement parts are two-weeks far in the future because they're travelling via snail mail or worse, relying on 2nd-hand spare parts off ebay.

edit: not to mention, the gigabit ethernet port is a bottleneck. you would probably hitting the bottleneck even by using four rotational disks.

If you're concerned about availability of parts, why not just keep some cold spares around? It's not like we're talking about expensive and specialized hardware here.

As a household backup device, rather than a NAS, this is actually pretty good.

Use mdadm RAID-10 across four spinning disks, or RAID-1 across a pair and have two slots free. If the board fails, you can plug them into any other Linux box with SATA ports available.

Most people have no better than 1Gb/s ethernet available in their house anyway. One expects a backup or restore to take a while, but also not to be interactive.

It's probably cheaper to repurpose an old desktop, but this will probably use less power, even when both are asleep.

ECC and zfs/btrfs I think are a must for any data you care about. RAID 1 isn’t going to save you when you can’t tell which copy of the given block is the corrupted one.

It completely depends on what your specific circumstances are, and something is always better than nothing.

(Also, you aren't getting ECC RAM at this price point, no matter what.)

What's a good use case for this? A house with a few laptops, a desktop and one server. Everything uses the server as their datastore. The server runs ZFS, but even ZFS can run out of luck, so you send over a snapshot to the backup device every day or three. It's insurance.

> Most people have no better than 1Gb/s ethernet available in their house anyway

This is partially true. Most people have no better than 1Gb/sec per single computer. That means that a single pc using the full 1Gb/sec would saturate the nas network I/O and degrade performances for all other clients.

Which may or may not be okay... You just know to be aware of that.

I keep saying backup, and other people keep re-saying NAS. These are different.

A NAS can serve as a backup.

Not all backups are suitable as a NAS.

Now, let me restate: Most people have no networks capable of significantly more than 1Gb/s between two hosts in their house. Complaining that a backup server is also attached at 1Gb/s is not going to cause people to discover that they have already purchased 10Gb/s switches, or LACP switches, or even acquire 2.5Gb/s switches.

Of the people who have 10Gb/s switches in their house, approximately none of them are going to look at this very low cost, low performance, low power system and say "yes! this is my new primary backup system!"

LACP switches aren't too bad. I picked up a pair of 16-port switches with vlan and LACP (1-4 ports) via web interface for around $100 combined.

2.5G or 10G switching is still too expensive for me.

If you aren’t using hardware raid or another disk management tool that locks you in to one manufacturer or worse one unique controller - you should be able to take your disks to any other machine and rebuild your array.

At least, that’s how you should be constructing a home NAS if that’s what you’re doing.

The same situation can arise with a popular/managed solution like qnap/syno.

I have a DS918 and really love it (it’s one of those set it and forget it machines) but I don’t totally know how it works. It’s Linux of course, but it’s sorta a black box.

So I think there is a lot to be said for DIY as long as you are aware of the drawbacks and engineer around them accordingly.

For the DIY way, I recommend the Helios64 if you're looking for an ARM SBC with a complete NAS package: https://kobol.io/

I have seen that, it's a really nice alternative.

If you absolutely need your data its not only on one device.

Here's what I want, but haven't been able to find:

I already have a beefy server at home ( it's actually a refurbished enterprise workstation, with 24 cores, but read on ).

However it lacks drives. What I ideally want is a a dumb drive bay I could buy, and then connect ( somehow ) to my existing server, so I could use its CPU and RAM.

I don't want this drive bay to have its own CPU, I just want it to hold data and transfer it over some wire. Ideally I would achieve close to gigabit speeds requesting data off of this server/drive bay.

Some options I've explored are those dumb quad storage bays that connect over USB-3. But I was worried about running ZFS over a USB interface, as well as potentially parallel read and writes with the quality of the USB controller.

Not sure what your workstation offers, but I bought myself an Icy Dock Black Vortex (MB074SP-B) [1]. It's just a case (internal 5,25", but that airflow! :D) which fits 4x3,5" drives with a 120mm fan in front. Slapped some noise damping feets under it. I then bought two low-profile eSATApd to internal SATA slotpanels [2] and 4 corresponding cables [3]. Works great so far!

And yes, you need eSATA, because the distance between cage and machine might be enough to introduce read/write errors. That's what I had, because I used normal SATA cables before.

[1]: https://www.icydock.com/goods.php?id=166

[2]: https://www.delock.de/produkte/G_61725/merkmale.html?setLang...

[3]: https://www.delock.de/produkte/G_84402/merkmale.html?setLang...

Man, SCSI daisy chaining did this, what, 30yrs ago.

Hm, rack mounted fiberchannel drive enclosures exist (~$100 used on fleabay, usually sans disk trays), but that'd require a fiberchannel card in your server, direct attach copper (or fiber) cabling, and driver finagling.

what you need is a JBOD!

Get a decent[1] external SAS adaptor, a SAS cable, and a second hand enclosure and bob's your noisy uncle!

something like this: https://www.bargainhardware.co.uk/dell-powervault-md1220-sto...

https://www.bargainhardware.co.uk/hp-z800-z820-external-mini...

Your workstation might even have a sas controller on it already.

This might be overkill, but even with spinny disks you'll be able to get fast random io[2]

[1] subjective. If yours is an enterprise workstation you'll most likley be able to get an official SAS controller for it.

[2] well about 10 iops per drive.

Use RPi for NAS is really a stretch. Just buy those low-power x86 mini-itx boards with 4GB memory and 4 SATA ports along with gigabit ethernet, they're around $150?

Note the smaller size of RPi-etc makes no sense when you're going to host 4 hard drives anyways which takes quite some space on their own, and you need a decent PSU for the drives too, and a solid case as well, etc.

Just buy those ASRock mini-itx boards at newsegg or somewhere and let RPi do what it's best at.

A quick google shows eMMC flash memory for the compute module...do you need to mindfully control writing to flash to keep something like this from nuking the boot device? Could be my choice of storage (whatever $4 buys at Microcenter) but the failure mode on my RPi weather station was SD card death. I'd hate to rely on that for a NAS.

Since some firmware update released in September 2020 it's possible to boot Raspberry Pi 4s from USB. It's a bit cumbersome to piece the info together, but it's there: https://www.raspberrypi.org/documentation/hardware/raspberry...

You can also boot the pi 4 via kind-of-special-sauce PXE. I wish they had just adopted standard PXE (why do we need a magic string? What the hell is going on with the three spaces in that magic string... Why can't we use the standard PXE way to specify the file to load?), but it's better than nothing. See https://www.raspberrypi.org/documentation/hardware/raspberry...

[edit: fixed spelling]

The 12V/5A power supply seems a little undersized for 4 x 3.5 HDD startup. I suspect that it's fine if the drives are started sequentially but the drives will probably want to pull more than 5A if they're all starting simultaneously. Possibly something to watch out for.

5A should be plenty for normal post-start operation.

Reply on news.ycombinator.com