The Wiretrustee SATA Pi board is a true SATA NAS | Hacker News Reader

The Wiretrustee SATA Pi board is a true SATA NAS

jeffgeerling.com

83 pointsgeerlingguy68 comments

With ZFS allowing for very robust data integrity on disk, the weak point in most budget NAS setups these days is going to be memory errors. It’s unfortunate that there are so few low-end options with any kind of ECC. The PCEngines APU2 has an ECC option, but it does not have the appropriate I/O for redundant storage.

If you're doing ZFS where that I/O bandwidth is a limiting factor, I'm thinking the 4GB of the APU2 makes it not interesting for ZFS in the first place (depending on the workloads of course, but you're probably going to want a bigger ARC).

I can see it work well as a sink for automated replicated snapshots, where I/O performance is not that critical.

Yup. Use mdadm to do RAID5, 6 or 10 and let it sit in your friend's place as a target for encrypted snapshots; do the same in exchange and you both have reliable offsite backups. If you need a big restore, go get the machine physically and bring it back to your network for full gigabit speeds.

unless you have several hundred TB, 4GB should be fine

it's quite happy with a 40tb array I have on a 2010 server with 4GB of ram

(and I've run ZFS on every laptop I've owned in the last decade or so)

ZFS works fine these days in low-RAM environments. As you say, you won’t benefit much from ARC, but for most NAS workloads that is fine.

The two problems I see are ZFS really wants a bit of a beefier machine, plus all the RAM you can throw at it. That pretty much excludes anything low end by default (not to mention more expensive ECC memory).

I think ZFS only needs lots of memory if you have certain features (notably, deduplication) enabled.

Meanwhile, you're still a lot better off, data-integrity-wise, with ZFS and non-ECC memory than you are with most other filesystems with non-ECC memory.

XFS also does checksums, but only for metadata.

It depends on the workload. With insufficent memory and/or latency, ZFS can perform orders of magnitude worse than ext4/xfs, even when tuned. (I speak from experience)

Not exactly the same, but did you ever try ZFS on a commodity USB stick? It can get really bad when circumstances are not in favor.

With not too much concurrency it can perform quite well even under constraints, though!

I've run ZFS on 2/4/8 & 16 gig machines. Yes, more ram makes things a bit faster, as you get a bigger arc cache.

however as you point out, 4gigs is more than enough to surface a 1gig connection. Especially on a bunch of SSDs

What's the solution for a budget NAS setup with ECC? Stick with Intel/AMD (for now)?

For that reason, my choice for a home NAS system board was a retired itx server board with a 20W Atom, 8 SATA ports, and ECC memory.

Comparable to RPi at compute power, but with vastly better I/O and, again, ECC RAM, because I don't want my data to have a chance to be silently corrupted.

I think there can be a niche for a system board like that, only open and based on ARM. IDK if any smaller ARM processors support ECC, though.

Do you have more details on that - part number or etc? I have an older beast of a machine that I'm thinking of replacing to get power consumption down, but I'd like to keep ECC for ZFS.

I just trawled eBay for "itx c2750". 2550 can also be considered for lower power, or 3750 for higher price and more capacity. Cxx58 parts can be considered instead of Cxx50 parts.

Of course all these CPUs are very old, 7-9 years since introduction. But it's ok by me, they are reasonably cold, have 4 or 8 cores, and I'm looking for a cold, silent, and reliable system, not a screaming fast system. At worst, a 10GbE card can be installed into the lone PCIe slot, but I don't expect my rig to saturate even the 1GbE built-in interface. (My board has four of these anyway.)

As of motherboard producers, SuperMicro makes the right kind: passive cooling, most / all ports wired, internal USB and / or m.2. Gigabyte made more consumer-oriented variants, but also serviceable.

ZFS doesn't need ECC ram to work correctly. This is an incorrect but commonly repeated myth.

See: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y...

But wouldn’t it work better with ECC?

The original comment did not claim that ZFS specifically needs ECC to run correctly.

Quote the OP,

> With ZFS allowing for very robust data integrity on disk, the weak point in most budget NAS setups these days is going to be memory errors.

You see, it didn't say "no ECC, no ZFS." It merely says: if your goal is to eliminate data corruption, once you've removed filesystem errors from the equation by using ZFS, the next term to eliminate is memory errors - which is entirely correct.

It doesn’t “need” it, but ZFS + ECC is a lot more effective than ZFS + high-BER RAM. Once you fix disk corruption, the next thing you’ll probably worry about is memory corruption.

> the weak point in most budget NAS setups these days is going to be memory errors

I use a HPE ProLiant MicroServer Gen8 as a (very) budget home/home-office NAS, I'm really not convinced that memory errors are its weak point(!)

It has the default (horrible) Celeron G1610T processor, was upgraded on delivery to 16G of memory, but has 60TB of storage - four internal SATA drives plus additional external USB drives.

I've tried and so far failed to get it to run stable 10GbE networking. Just sourced a new (old) ConnectX-3 card from eBay to try again...

I use an i3-4370 with 32G of ECC in a Supermicro 4u case as my ZFS NAS - I got it new for the purpose and it's still running strong seven years later. The only changes I've made have been doubling the RAM from 16G, adding in new storage as needed, and adding a dual 10gbit card when I upgraded my network.

I tried a couple of 10gbit cards, but ended up getting a half dozen used Intel X520-DA2 cards for dirt cheap. I've not had any issue with these, and my NAS can easily saturate at least one link. The cards have been reliable in every machine I've tried and can be found for $75-100 - not too bad for dual port 10gbit!

As a side note, I was surprised to find that i3 supports ECC.

I use a MicroServer too (an older one). They have ECC memory, so you should indeed be fine on this point. But price and electricity consumption are much higher than a RPi.

I agree that the CPU is the weak point (mine has a turion n40l which is slow as hell).

> mine has a turion n40l

Heh, I have two of those older models lurking in the garage from some older experimental stuff.

They were at one point configured as a DRBD two-node cluster, which was fun, although performance was dire.

I've run ZFS on my personal (read: cheap) hardware for several hundred machine years and never had a problem with standard RAM

(and issues would be detectable with the monthly scrub)

obviously ECC is better but you're no worse off than you would be with mdraid or some awful hardware controller

> and issues would be detectable with the monthly scrub

Depends where the corruption happened. Scrub can’t detect data that got corrupted before the checksum was taken.

A ryzen system with ecc can be had for less than $400 build your own.

yes, but even the smallest ryzen sucks down more power and the system is physically bigger.

but its way more powerful so can do more stuff.

Yeah low power single board computer systems with all the right features tend to be very expensive and difficult to source without large orders. A ryzen system at 20 or 40 watts idle is something you can probably put up with for $400

I completely agree with you. ZFS has led the way in data integrity and ergonomics, and it's time we got it in a small form factor with reliable RAM.

Many commenters have stated that it would be better to go with a Ryzen/ECC or some server-grade Atom board. While that may be true from a performance standpoint, it is not true if your priorities are low power usage and saving space.

I have several big beefy servers to run ZFS on my network. What I am missing is something tiny and quiet that can do the same job at a small scale just as well while the big machines are turned off.

I am not concerned about maintaining fast I/O or running some RAM-hogging zraid6. I just want 4 separate SSDs running 4 separate spools that I can read/write to anytime and later sync with the big machines when they're online. The even the 4GB RPi would perform that fine - I just don't see a point if the RAM is not ECC.

> low power usage

How about turning the NAS on-and-off as you need it? There's no need to run an "always on" computer unless you want to.

If you want to run an IRC bouncer or a webserver, then buy a Rasp. Pi for that. The NAS is primarily storage, which doesn't necessarily need to be always on. (Besides, turning the computer off is the ultimate security move. You can't get that data hacked if the computer isn't even on)

------------

> I have several big beefy servers to run ZFS on my network. What I am missing is something tiny and quiet that can do the same job at a small scale just as well while the big machines are turned off.

Ever try just... sticking multiple small-and-cheap hard drives into the clients?

Its not too hard to make redundant RAID1-like storage spaces (Windows-clients) or RAID1-like LVM (Linux-clients).

> the weak point in most budget NAS setups these days is going to be memory errors

In practice, the weak point in most budget NAS setups (1) no backups "because RAID" and then getting hit by a cryptolocker (2) drives dying

> (2) drives dying

Is that really a problem?

I'm still able to find 1TB new hard drives at my local microcenter for just $30. So its not really that expensive to set up a 4x drive RAID10 or RAID6 array.

4x 1TB in RAID10 or RAID6 == 2TB with redundancy for $120ish. Go up to 4TB drives ($100 each) and you'll get 8TB capacity for maybe $400ish.

Prices suck right now, even for hard drives. (I used to get 6TB hard drives for $150... but the 4TB point still seems reasonable).

> (1) no backups and cryptolockers

That seems to be easily solved by regular ZFS-snapshots.

Hard drives do die, and while many (possibly most?) deaths are random materials failure, some deaths are correlated to batches and some are firmware problems which are often correlated.

Avoiding these is hard. Ideally, you want to get drives from different batches and offset power on times by about a week per drive. Assuming you will notice a failure, obtain a replacement, and replace the failed drive within a week. If that's not realistic, you should wait even longer between drive power on, which makes setup a very long process. It's also hard to ensure drives are from different batches unless you buy in person or get different brands, but there's not four brands to buy from anymore.

It's a little easier to do best practices when you replace drives (to get more capacity for example)... Buy one new drive once every other month, and you probably will get different batches, and you'll most likely have sufficient time to replace drives without running into the same firmware bug on all the drives.

If only all these mega corporations that get hit by crypto lockers had heard the magic bullet of ZFS! /s

If any corporation has a proper and secure backup, then crypto-lockers are simply a null-and-void issue. You get cryptolockered, then just restore from backup and you're fine.

For online backups: ZFS snapshots ought to handle the common cryptolocker case (where cryptolocker infects the client machine and encrypts all the data). However, if the virus stops there, the ZFS snapshot (which is controlled on the NAS) can restore your data from the last snapshot no problem.

I'm sure that criminal gangs are willing to go one step further and maybe attack the NAS and delete the ZFS snapshots, but its still a 2nd step that must occur before you have a significant failure.

---------------------

Its not a magic bullet, but its a significant defense mechanism that seems to handle the current, common case.

This but unironically. Sometimes I find a technology or strategy where I think “people are insane for not using this”. ZFS is definitely one of those. It reduces so many IT costs by orders of magnitude. (Storage requirements for denormalized data, backup time and bandwidth, frequency of backups, procedural cost of restoring from backups, cost of dealing with disk failure, data corruption costs, etc.)

ZFS backups are easier than anything else I’ve used. I just zap incremental snapshots over to an rsync.net ZFS account.

For drives dying, I just ordered the 3 drives in my zpool from 3 different websites (Newegg, Amazon, and someone I forget) so they’d be different batches.

> ZFS backups are easier

the weak point in home based geographically unique off-site backups (like, put a small zfs box in your family member's house) is that while the ordinary residential internet user might have cablemodem service with 500 Mbps down, you may also have only 16-20Mbps up. Good luck backing up 15-20TB over 16Mbps upstream and possible 1TB/month quotas from an ISP.

Reality here is lucky to have 50MBps down and 7MBps up and I am in the fourth largest city in North America. More likely 30/3 with actual throughput being 20/1.

If you don’t have enough bandwidth to do incremental ZFS snapshot backups, you don’t have enough bandwidth to do any kind of backups. ZFS incremental snapshots are more efficient than anything else I’ve used, e.g. rsync.

> The PCEngines APU2 has an ECC option, but it does not have the appropriate I/O

from looking at https://pcengines.ch/apu2.htm , perhaps one could bridge from the miniPCI express to a SAN eSATA tower? (rough example: https://hobo.house/2016/02/03/replacing-failed-linux-raid-di... )

Ok, I see this a lot. Does anyone have real world experience with meaningful corruption which could be pointed at memory errors? A certain sort of person is very interested in ECC and I have a lot of doubt that it actually causes anybody trouble commensurate with the attention it gets (unless you are Google, and you aren't Google)

Even if a few bits are flipped in a media file once during transfer or even permanently... or if the OS state of my storage box gets corrupted and I have to reboot it... so what?

RAM BER is shockingly high, like several bits per GB per year. Dozens or hundreds of data corruptions per year does not meet my requirements for even a home NAS, even if most are transient.

Well what are your requirements?

What are the outcomes you think are going to happen which you are trying to avoid?

The "error rate" of, say, a dust mote on my TV screen would seem to be orders of magnitude higher than a few flipped bits per GB*year. I have quite a bit of doubt about how many bit errors would ever hit A) during a time in which I am actually retrieving data (or much much less likely saving) and B) actually hits the surface the data is resident in memory and C) has an impact that raises to a perceptual level.

This is really cool, except for not being available yet. The most interesting available in this category and price class I've been are otherwise Odroid HC4 (2xSATA + USB2.0 + microSD. I have one for backups) and RockPi's 2/4/5 SATA hats.

One step up in price and Kobol Helios64 looks really nice as well (also unavailable, though).

Everywhere I look lately, it seems like things are not available (or if they are, for a bit of a markup—I needed a spare 8 TB hard drive, and the price is up about 30% from a couple months ago!).

I hope this current component shortage and crypto bull market both end soon.

Yup, same. I really don't think the crypto mining/chia are as much to blame as people make it out to be here, though. For GPUs it's absolutely been a major factor, but the general component shortage combined with supply chain issues caused by the pandemic are more likely the cause I think.

Maybe specifically for SBCs and NUCs, I can be wrong and people running their own nodes are actually making a difference. I doubt it, since manufacturers and vendors are all citing shortage upstream rather than increased demand, but if that's the case I think That's Just A Good Thing, setting the stage for more digital independence and decentralization in general.

For this type of application, check out used HP t5xx or t6xx thin clients. They are great little boxes that usually have 2 M2 slots and USB3.

Usually they are <$100 and better all around than a Pi like device for this application.

For this application? Even with adapters to connect m.2 to the desired spinning rust drives, which probably can't be used with the case closed, that only gets you two SATA connections, not four.

The argument can be made quite well that there are better alternatives to the Pi for a low-end NAS, but the HP slimline thin clients don't look like it.

For home NAS?

USB3 connected enclosure and/or an internal SSD is a pretty great solution.

IMO, if you’re building more than that, it’s kind of silly to limit it to Pi-like price point.

USB storage tends to disconnect aftera few days on Linux (tested with various USB-powered and mains-powered HDDs), and the only way tho resume is physically disconnecting and reconnecting the device.

This is fantastic, thank you!

Ughh... I was just about to buy an Odroid HC4, but looks like they're out of stock like so many other things right now.

To me the issue with RPis storage is the lack of hardware encryption. No matter how much you increase its I/O capabilities it won't be able to use disk encryption without having to rely on CPU cycles.

I'm still surprised that they did not include a crypto accelerator on the Pi4. It seems like a very low cost peripheral, with a high performance impact for many use cases (drive encryption, network encryption, etc...). I ran some benchmarks between a Pi4 and a Nanopi M4 and it wasn't even close.

Did you try different ciphers? Whilst there's no AES acceleration there are other ciphers that can leverage the hardware better. i.e. ChaCha20

Not really, I mostly cared about AES which was around 25x faster. Here's what a NanoPi M4 does:

  Cryptsetup
  #     Algorithm |       Key |      Encryption |      Decryption
          aes-cbc        128b       465.5 MiB/s       654.0 MiB/s
  
  OpenSSL Speed for ChaCha
  type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
  chacha20-poly1305    85242.48k   144024.55k   265936.64k   309418.33k   317977.94k   317658.45k

vs Pi4

  Cryptsetup
  #     Algorithm |       Key |      Encryption |      Decryption
          aes-cbc        128b        18.9 MiB/s        31.9 MiB/s
  
  OpenSSL Speed for ChaCha
  type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
  chacha20-poly1305    24833.53k    55555.22k    90574.34k    99022.17k   102659.41k   100592.30k

Could always try AES-Adiantum for storage encryption. https://www.raspberrypi.org/forums/viewtopic.php?t=275542

I keep hoping some day the drives will have their own networking built in. Kioxia, a Toshiba spin off, announced a network-attached NVMe-oF drive last September[1], and I seem to recall one of the major drive players had similar intents a bit back... ah yes, the Seagate Kinetic drives with dual 1Gbit[2] & an object storage OS built in to the drive. These days Seagate seems to be pushing a software platform CORTX[3], which I hope some day perhaps has hardware products too (but right now seems to be for classic linux-based network appliances)

Ideally we start using 5 or 10Gbit ethernet for these cases. We could continue to treat these drives like they are direct attached, even though they are network attached, and either have one computer running RAID, or have Ceph and a bunch of computers running it's distributed system to tap the drives.

Ideally though, we need new clustered file-systems, where any computer can read the drives. That is, I'd guess, a long way off. Legacy devices (home media players) would need to go through some kind of legacy gateway.

[1] https://business.kioxia.com/en-us/news/2020/ssd-20200922-2.h...

[2] https://www.snia.org/sites/default/files/MayurShelty_Seagate...

[3] https://github.com/Seagate/cortx

I don't think the Pi's cpu has any sort of AES acceleration instructions (such as aes-ni on intel), which means that doing FDE is likely going to bottleneck this pretty badly.

This is not available. Someone (thanks, jeff!) just posted this in the comments:

"The day I finished editing the video for the Wiretrustee SATA, the folks behind the board announced they were postponing production because of the component shortage—some parts, most notably the SATA controller chip, had increased in price dramatically, and would only be available in small quantities."

Just more rPI vapourware. Maybe repost when someone can actually get their hands on this?

I think it's not fair to hold against them something that's affecting Ford and Apple too...

Are you under the impression that if I walked into an Apple store or a Ford dealership today with cash, that I would not be able to leave with a new laptop or car?

Depending who you try to order it from, you may be backordered for a month or two trying to order an iPad, yes. Ford has shut down some of their factories because they can't get their hands on parts.

For very large companies, stock still exists, especially if built over the last year or two, but as shortages continue, in the near future, you may, in fact, not be able to leave a Ford dealership with a new car.

That's all very different from "you can't buy this, right this second".

So close to what I want. I want something I can use as a "home server", with a priority on having as much control over the hardware as possible. Ideally, it would be an interesting project to design one, but I suspect that the result would be exactly like the rpi (but badly executed).

I would buy this in an instant once someone posts a review for a complete unit. This fixes the main problem I have with the RPi which is crappy storage support.

Very cool! There’s lots of great business ideas that could use something like this.

Did anybody else skim the headlines and wonder if a board of trustees had really been replaced by a NAS?

Reply on news.ycombinator.com