Hacker News Reader

> Not a great time price-wise to be building a NAS

That under-states the matter. It is a terrible time, price-wise, to build a NAS.

I'd almost rather have no AI whatsoever and have storage 1/10 the price of pre-AI times.

(If there were a magical choice between having AI and significantly more expensive storage, and having no AI and some program to dump that investment money into getting and somehow leveraging significantly more available storage, that is.)

I would kill (figuratively, anyway) to have no AI whatsoever. No slop machine threatening to replace my job, or turn my job into babysitting its stupidity, and hardware would be reasonably priced? That would be awesome. AI has brought me nothing but downside.

But we now have trillionaires, aren't you happy about that? It means you might be the one some day too /s

I immigrated wanting to become a millionaire. Never found time working my ass off. Now that I get pension and tokens from Musk and Zuck, I can finally try!

AI has freed most people I know from the tedious job of writing actual emails and birthday greetings - leaving then more time to e.g. wash dishes or clean the floor.

Seriously, if arts and creativity is what sets humans apart from other animals, then AI has almost completely displaced our capacity to even consider doing these activities ourselves. People reach for AI when they should be composing a birthday greeting themselves.

Is it a genuine birthday greeting if you just let AI do the job?

Was it a genuine birthday greeting when you bought a card? Or used an online card? I don’t really see the difference. The point is you thought of them, not the actual words

With AI you don't have to think about them.

Great business opportunity there. Make a free service that sends AI generated birthday notes to people. All the user has to provide is the other persons birthday.

People cannot be relied upon to provide accurate birthdays but nobody would suffer the social faux pas of an incorrect birthday congrats note. Nor would they send it to a spam catching email, but rather are guaranteed to send it to a regularly checked address. They know the persons birthday after all.

You can then sell high quality birthday information correlated to contact information to ad agencies.

Fuck this current internet. So dystopian.

Then the receiver AI can thank the sending AI before deleting.

that was their point

I used to write birthday emails to my friends years ago. I quit when calendar systems started automatically promoting me when birthdays happened. Remembering someone's birthday no longer seemed like a significant effort on my part and so it wasn't worth the bother since it didn't say as much.

For 90% of birthday greetings, are they ever genuine?

Like seriously, outside of some close friends and family, are you sitting there deliberating over a message for your coworker Steve in the Slack chat or a cousin you barely talk to outside of a birthday and Christmas card?

People were Googling these things. Those who were good with words, didn’t do that before AI and won’t do it today. They don’t have to.

Are you being sarcastic? There's no way you believe people couldn't wash dishes or clean floors because of all the emails and birthday greetings they had to write, or that people have almost completely lost the ability to consider making art.

I believe it was rhetorical. They were trying to illustrate that we've automated tasks that people might find to be fulfilling (drawing, writing) but are still a long way off from affordably automating drudgery.

> or that people have almost completely lost the ability to consider making art

They haven't lost it, no, but there's a lot of financial incentive to stop paying people to do it.

It's a paraphrasing of the popular 2024 Joanna Maciejewska quote: "I want AI to do my laundry and dishes so that I can do art and writing, not for AI to do my art and writing so that I can do my laundry and dishes."

someone at work unironically said they use ai to compose "thoughtful responses" to people, and my immediate thought was that what they are doing is the literal opposite of thoughtful

I'm doing it anyway and I've found some secondhand deals. Maybe COVID homelabbers are offloading gear?

I built a 24 TB HDD NAS in a 36 bay chassis for around $1000 all-in: mb, ram, rails, chassis, rack, disks, hba, nic

Awful Watts/TB but the plan is to run it and the GPU rig on solar.

I obtained a used Dell PowerEdge and just stuffed a bunch of drives in there. It's old enough that the RAM is not overly expensive (DDR3), so it has 192GB of RAM to effectively cache the data, and then 32TB of raw storage. (Of course I regret not spending the extra money to take it to 384GB now.)

The nice thing about the array being big is that I can just RAID 1 it instead of worrying about tinkering with RAID 5, and leave a drive as a hot spare.

PCIe 2.5G NIC for the uplink, and then it can serve over SMB or iSCSI. The main use case for this thing, incidentally, is it just is a caching proxy that holds Docker images, models off of Huggingface, and so on.

36 bay? Why so many?

The 36 bay was all I could find on fb marketplace.

I fell in love with the silly idea of squeezing 10G NIC + 10G writes (to a 2x12 RAID 10 HDD array) on the thunderbolt bus of my 12 year old macbook pro.

Then I got distracted with the server mb and truenas and ZFS.

36 bays and only 24T of storage?

Can't you get that in a single disk that costs less than a kilodollar?

That will be a single point of failure, though. (Which you might not care about depending on the backup strategy.)

What size discs are you using? This post is kind of hilarious, 24TB is achievable in literally a 2-bay NAS nowadays.

> I'd almost rather have no AI whatsoever and have storage 1/10 the price of pre-AI times.

Almost? ALMOST!?!?!

If you handed me a button that would make it like LLMs had never existed I'd be slamming that button so hard Sam Altman's clothes would spin around.

Return memory and storage prices to normal, undo the sloppification of ~everything, remove all these annoying "features" that are so useful they have to force them upon people, and make scammers actually have to put in a slight bit of effort, all at the "cost" of real human developers, artists, writers, etc. getting paid for their work....

If this is a hard decision for you, you are the problem.

Same. Every time I think about this, I come back to the fact that I was excellent at my job before AI, and even enjoyed aspects of it, so I would be fine if tomorrow it all blew up.

Hey everyone, stop progress! Wolrah has decided that November 29, 2022 is as far as things should go. Because everyone knows before that day, there were no scams, every artist lived lavishly from commissions, and software was flawless and without any extraneous features.

There's a difference between standing in the rain and being hit by a tsunami, even if it's both just water.

That's quite uncharitable. What are you trying to achieve here?

They (probably) don't want to stop progress (especially unqualified like this) in general. They'd like a world where LLMs didn't come to exist.

And whether LLMs are progress at all remains to be proven.

Same here, the usefulness of the slop machines is quite limited compared to the downsides.

Oh come on! I'll take a few years of expensive RAM in order for humanity to get wide access to something near as makes no difference to the Star Trek Ship computer.

Which is 100% what this has felt like to work with, all spring and summer.

So we're in a slop phase, it'll pass. The first few years of youtube gave no hints that we'd get stuff like Veritasium.

Ignore the slop and use the tools to create something you never thought you could do. That's what it's for!

Same, I'd smash it like it was offering me a million dollars. I cannot think of a single positive thing that has come out of the AI boom. It has been a net negative for all of society.

I don't really agree with the specifics but I can identify with the emotion. To me the problem with AI isn't AI, it's that investors are idiots and put hundreds of billions of dollars into AI companies that have near zero chance of being profitable enough to pay them back and those companies took that money and blew up computer parts supply chains that are vital to the rest of human existence. If we didn't have this massive build out that has no chance of repaying itself and instead had a build out that only doubled ram/gpu/nvme prices we would be far better off. As it stands, we're going to have a ton of gear that will be of questionable utility once any of the disruptors get their 100X+ per watt improved/different hardware going and to get that hardware we broke everything else. This ai stuff will also likely plateau and have the openweights models and opensource tooling catch up and at that point the companies are basically GPU REITS with diminishing demand as the stuff ages out and the disruptor's gear gets built for much better performance per watt. They've made it really ugly for us now and it will get really ugly for them in the midterm future imo. Not a good outcome for anyone.

You're telling me.

I had a drive go bad in my 4 year old home-built NAS a few weeks ago, and it cost 2x what it originally did to buy the same capacity drive, and that was going to a grey-market importer on eBay, it was more on "reputable" sites.

Looking at current hd prices, I keep kicking myself for giving away several 3tb and couple of 4tb drives I had in my unraid server few years ago when I started upgrading my array to 8 and 12tb drives. At the time, a drive slot in my server was more valuable to me than a mere 3tb drive. i could have kept them for emergencies if any one of my array drives fail.

I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real?

Like I’m already giving up two full drives for redundancy (which saved my ass - I recently had two drives fail on me in quick succession — both SSDs from what looks like an identical batch) but then the advice is kinda saying I need to keep at least another drive worth of space free for the pool to perform well and not crap itself. That hurts with current prices for sure.

ZFS will auto-degrade performance if there isn't enough headroom. In addition, if you use SSD, you also want the headroom, because otherwise you end up writing and rewriting on the same small empty space which kills SSD (unless you have enterprise SSD, which have built-in headroom).

Not sure why I'm being downvoted for being factually correct (and also, why I can't edit my above comment).

ZFS will definitely degrade write performance gradually from 90% utilization, and will hit a stronger cliff at 95%+. Same with SSD, using a consumer SSD above 90%+ utilization would rapidly degrade its lifetime. The effect will be smaller for very large pools and very large files, but the effect is stil there.

I'm repeating this so that people who set up these drives know what is actually going to happen.

> Not sure why I'm being downvoted for being factually correct

Maybe because your statement about SSDs is incorrect. Consumer SSDs have spare blocks too, and wear leveling prevents the scenario you're describing.

I appreciate the reply, but my warning consumer SSD is not incorrect. Perhaps I should have clarified that While they typically have a small overprovisioning of 7%, enterprise SSDs have close to 4 times as much, exactly to allow more writes (which is very relevant if running a ZFS raid on the drive). Keeping 10-20% headroom will prolong your SSD's life significantly, and increase performance of ZFS.

Go ahead and let them try consumer SSD's. It seems to be a lesson everyone has to learn with ZFS once.

> I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real?

I'm not entirely sure, but it seems to me that free space (and the 20% reservation) is mostly a proxy for fragmentation, and you can therefore better look at fragmentation directly. That would mean that if you mostly store large files, there shouldn't be a lot of fragmentation even at high utilization. The whole "ZFS changes allocation algorithm from 80% usage on" is something of 10+ years distant past, and lots of things around the allocator have been improved. It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated.

> It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated.

Well, other filesystems can defragment.

I don't think they are. I've already filled a ZFS pool almost to the brim with no adverse effects.

My experience has been escalating fragmentation past 90%, along with lower performance owing to the fragmentation.

For home usage, if you have backups raidz1 is fine (just do an incremental backup at the first sign of trouble). If you don’t have backups, then you probably shouldn’t be running a NAS in the first place.

This isn’t black and white. You might have huge amounts of non-essential data that aren’t worth the cost of off-site backups, but worth the cost of an extra disk of redundancy to lessen the risk. Even when you do have backups, it will reduce the risk of extended downtime (and possibly egress costs) caused by having to restore large amounts of data from backups.

For most people snapshots are enough backup. It is still useful to have an offsite backup, but realistically fire is rare enough that you can risk it, and that is about the only risk most people have (you can have your NAS in a location likely far from where a fire might break out to reduce the risk farther).

Fire, malware, accidental deletions, capricious RAID controller (pre-ZFS). And that’s only the stuff that happened to me. Add power surge, theft, correlation in SSD failures (eg power on counter overflow firmware bug), damaging the array while moving, etc.

Malware, accidental deletion, RAID controller issues - all things ZFS with snapshots are immune to. You can decide if the other worry you or not - the risk is not zero, but it may be acceptable. Your backups are all subject to similar issues.

Snapshots don’t protect you from malware. I bet 99% of home users use the same credentials on all their machines, once a malware compromised one, the others are compromised within seconds.

[edit] also snapshots aren’t really workable for large files. Remux a movie file and now it occupies twice the space.

The middle ground I go with is to structure the pool using datasets and zvols, and then periodically backup the critical ones using zfs send. For example, I can live without my movie collection, but I cannot afford to lose personal photos.

In the Linux ISO community the math is different because none of the data is essential, however there is a lot of it. Backing it all up even one time to an off-site location would be a 2x expense for a hobby. So 3-2-1 is hard to justify.

Instead the math can be different e.g. it's ok to lose some data but not all. Therefore you might prefer the unraid approach over zfs where losing more than parity doesn't kill the whole pool.

I'm actually in this boat right now. I have 2x26TB drives in a 4 base nas (two empty, plus 3x M2 empty) and I'm trying to decide the best way to set them up. I have around 3TB of backups that I care about as backups (but they don't need to be online, just archive backups at this point), and around 5TB of media that can be replaced.

I don't want to lose 26TB just for a mirror, and I have a spare 8TB USB HDD. I'm torn between unraid and just Debian, and I'm torn between just two separate devices and one RAID 1 partition one RAID 0

The risk of z1 is that if you get a read error during resilvering, that data is permanently corrupted. The odds of this happening go way up the larger your individual drives are. This is why I chose RAIDZ2 for my NAS. I've had to resilver 2-3 times over the past 10 years, and never lost a byte of data.

> The odds of this happening go way up the larger your individual drives are.

This is a common claim, but honestly, citation needed.

We can't just apply bit error rates from the datasheet that haven't been updated in 15 years. I'm sure a 2-day rebuild is a little more risky than a 4-hour rebuild, but I'm not convinced it's by all that much. Especially if you had a monthly scrub going to prevent disk rot and disks secretly getting super fragile.

> I've had to resilver 2-3 times over the past 10 years, and never lost a byte of data.

And how many times did you have a read error on one of your other disks during a resilver?

>> The odds of this happening go way up the larger your individual drives are.

> This is a common claim, but honestly, citation needed.

FWIW, here's the quick & dirty argument for that:

Assume:

- We're working within a given generation of disk technology, with comparable drive and interface tech.

- The drives' I/O speed is relatively constant, not scaling with the drive capacity.

- The chance of any given bit (of data on a disk) going bad, in a given time interval, is relatively constant.

- The drives, regardless of size, will be some fixed-ish percentage full. (If nothing else, the boss won't sign off on overspending for unneeded capacity.)

From there - the time needed for resilvering scales linearly with the drive size. As does the number of non-redundant bits that could go bad in any given second of the resilvering. Linear time linear - so the chance of data loss during resilvering rises with the square of the disk size.

> - The chance of any given bit (of data on a disk) going bad, in a given time interval, is relatively constant.

We can't just assume this, though. Especially for data that's being checked regularly, because the drive can see how many pre-ECC bits have gone bad and rewrite iffy sectors.

And even if it's true, it's normal to go years without errors. If you have a 15% chance of getting an error each year, and your degraded array has 4 drives without redundancy, then the chance of getting an error in 4 hours is .02% and your chance of getting an error in 48 hours is .3%. Having that risk show up once every few years is pretty negligible compared to everything else that could go wrong with your array.

On top of that, if you don't have a hot spare and it takes 5 days to get a new drive, the difference in resilvering speeds is now 5.2 days versus 7 days and that barely matters.

The main situation where you have to worry about extra drives failing is when you think the resilvering process will greatly accelerate drive deaths. And that depends very much on the specific numbers you have in mind.

> This is a common claim, but honestly, citation needed.

This is based on the data I had available 10 years ago when I set up this array… If you have data that points to drives having fewer read errors per GB in recent years, I'd love to see that citation.

> And how many times did you have a read error on one of your other disks during a resilver?

I don't know, because I have RAIDZ2, so that read error would probably be corrected without it being reported to me.

> This is based on the data I had available 10 years ago when I set up this array

Data you collected or data you found? Any idea where it was?

I've only seen oversimplified math, not real data, and different versions of the math make different assumptions.

> I don't know, because I have RAIDZ2, so that read error would probably be corrected without it being reported to me.

Zpool status gives you a per-disk error count. I kind of assumed someone doing a resilver would use that command.

> Data you collected or data you found? Any idea where it was?

I think it was a lengthy blog post citing manufacturer data, found via the FreeNAS forums. The added safety of RAIDZ2 was a good tradeoff. Someone else on this thread reports they lost two SSDs in rapid succession, which I hear happens sometimes with drives from the same batch. In any event, I felt no need to launch an independent investigation of the math and underlying assumptions.

> Zpool status gives you a per-disk error count. I kind of assumed someone doing a resilver would use that command.

Yes, I did indeed use zpool to do the resilver… I don't recall if there were any read errors on other drives at the time of those failures, but probably not, as the scrub probably would have flagged it. They've all been replaced with higher capacity drives, so at the moment I see "No known data errors"

Okay, interesting. Also to be clear I do think Z2 makes sense in a lot of situations, but I suspect that it's not very dependant on drive size like a lot of people think. If I wanted high reliability I'd use Z2 even with small drives.

Also to reply to your other comment: Personally one extra drive is a small price to pay for the peace of mind.

I achieve peace of mind via backups. For each copy, once I get the AFR below 2% I'm not too fussed about pushing it lower.

> The odds of this happening go way up the larger your individual drives are.

Technically true.

Practically? There are several other major factors - such as the quality of your drives, whether you periodically run zfs scrubs and/or disk read tests, and how long a resilver takes to complete.

Behind those are the eternal duo - size of budget and cost of failure.

Or like me where you get two SSDs in your pool that happen to likely be from the same batch and both decide to permanently disappear at roughly the same time :(

> So now I have a large script which debootstraps a Debian based NAS into being.

Is part of this publicly available? After seeing, - https://words.filippo.io/frood/ - https://0pointer.net/blog/fitting-everything-together.html

I've been looking at setting something up with mkosi/systemd for my home server/NAS - it's interesting seeing how different setups are solving the problem for inspiration.

> combined with LUKS, TPM keys

Does it work? Server can reboot and use TPM to unlock rootfs? What about /boot - encrypted and tamper proof? Resists evil maid attack?

Rabbit hole, I know, but so fascinating if you solved it all :)

If you are worried about someone breaking in to your house and replacing the bootloader while leaving your drives in place I probably wouldn't use the TPM auto unlock even if in theory secure boot should be able to handle this.

But in reality that will never happen and the only actual attack you need to be worried about is junkies breaking in and flogging the drives on facebook marketplace. For which, this level of security is fine.

> junkies breaking in and flogging the drives

For this you dont need TPM. Just a LUKS key in rootfs /etc. He said TPM :)

I run my cheap hosting box with a cleartext boot setup that uses ssh to automatically grab the key for the real root from my home server (or an alternate at my MILs house). Using FreeBSD, but similar concepts.

A previous hoster once gave me someone else's drives without wiping them. I don't want random customers snooping around on my data if a similar mistake happens with my disks.

For home use, I run without disk encryption. If I ever need to do data recovery, it's not going to be possible with encrypted disks and one point of a centralized NAS is to have stable long term storage.

> What about /boot - encrypted and tamper proof? Resists evil maid attack?

> Rabbit hole, I know, but so fascinating if you solved it all :)

Encrypting `/boot/` is the wrong thing to do. One can just replace it with something that logs the encryption key instead (and sends it off to somewhere). You actually need to verify that what you expect to be there is actually there and a encryption isn't gonna do that for you.

A reasonably secure boot chain looks roughly as follows: bootloader is checked by the UEFI (read secure-boot), bootloader chain loads a signed UKI which also uses secure boot verification (or just skip the bootloader if you only have a single kernel with no fallback), this then loads a signed dm-verity `/usr/` which brings you to the login screen. Root is just data (e.g. mounted `noexec`) and is encrypted using e.g. PCR 11 so that it only unlocks if the UKI is the one that is expected (with some other backup unlock method).

This is basically #2 of the "Design Goals" of Poetterings "Fitting everything together"[0]. It's a good read if you are looking to designing your own Linux image and are looking for some inspiration. There is ParticleOS[1] which somewhat dogfoods this kind of system, though I don't think it is anything remotely considered for a production system.

[0]: https://0pointer.net/blog/fitting-everything-together.html

[1]: https://github.com/systemd/particleos

> 4x the 14 TB WD Elements Desktop.

I recommend staying away from SATA drives (huge consumer rush) and look for SAS drives on eBay, particularly HC520 Helium drives from HGST/WD. Need a SAS3008 PCIe adapter ($20) and a SFF-8643 splitter ($30). No backplane is required. Huge lots of decommissioned drives frequently appear. I bought Qty 10* of HC520 (12TB) SAS drives for $1000 about 3 weeks ago, avg age is about 2.5 years, still well within its rated lifetime.

Yea may be some of the stuff is fear mongering and cargo culting. I was told ECC is necessary for ZFS. When the article was written, it was cheap af (2024) to buy ECC RAM so not much consideration was given to it.

-- (*) I have no idea what to do with it. Anyone has any good idea for using 120TB space? I have about 40TB unused bandwidth in the datacenter, may be host a Debian mirror? Donate storage/bandwidth to Internet Archive? Please contact me, appreciate it.

> I bought Qty 10* of HC520 (12TB) SAS drives for $1000 about 3 weeks ago, avg age is about 2.5 years, still well within its rated lifetime.

That's cheap indeed. Enough headroom for some failing disks too. How is the noise and power usage? I didn't look at SAS drives at all, because my impression was that they're very noisy. I can place my NAS in a closed off room, but it's not too far away and I was afraid SAS drives would be audible through the wall. At the same time, the shucked drive I'm using presents itself as an WD Ultrastar, which comes very close to a SAS drive, and isn't very noisy.

> How is the noise and power usage?

I think they're same as SATA drives, just different interface. AFAIK they have the same physical dimensions and same internals.

This reply is exactly right about using enterprise drives (hdd), and ideally SAS. Note the power usage of SAS vs sata for an identical drive make/model is very small sometimes 0.5 W to 1.5 W at idle (idle power draw is what matters for spinning HDD’s as underload it doesn’t increase much and idle is what you’ll be at 98% of the time). Also enterprise SSDs are much better than consumer SSD‘s for ZFS, and be aware that when you get above the 4 TB size SSD drives frequently use more power than HDD drives (i’m mainly referring to enterprise non-consumer level drives as that’s all I run for 100s of disks across many zfs systems over past 15 yrs).

Also something people forget for home nas or figuring out cost for a nas, the power draw of the entire system times your electricity price you then have to multiply this times 2x to 4x times if your climate requires cooling air conditioning of any kind

I've been thinking about building my own NAS as well. Mind sharing how much did you pay for those hard drives, and what motherboard did you choose?

It was €329 per hard drive (through a reputable store), and I chose the ASRock B650M-HDV/M.2 mATX motherboard combined with a Ryzen 5 8500G. Stock CPU cooler, replaced the Jonsbo case fans with Arctic P12 Pro PST LN. I only slightly regret the PSU (MSI MAG A650GL), which could be quieter. Not that it's very noisy, and it's a great PSU otherwise, but I should've chosen one that just shuts the fan down at low power usage.

https://diskprices.com/ is great for this

You have SATA or SAS to pick from. The CPU requirements for a storage server are not high. On a typical ATX board you have motherboard SATA and can put SAS controllers in the spare PCIe slots.

My first "NAS" was two 22TB hard drives in a ZFS pool on my motherboard SATA

ATA (SATA), SCSI (SAS), and NVM (NVMe).

no. just sata & sas. There is no nvme spinning drive and flash loses bits just from sitting, it's not archival. flash is good for working not for storing.

A NAS is plugged in. Flash works fine for it.

This is true, but if you use a file system that does scrubs (and you should use such a file system for a NAS), and if you keep the NAS on 24*7, this won’t be an issue.

There should be more sites like diskprices. So refreshing to see no styling, no cookie banners, just pure information.

Yep. The 4 Samsung SSD 990 PRO 4TB NVMe SSDs are $1100 apiece. Yikes!

https://www.samsung.com/us/memory-storage/nvme-ssd/990-pro-p...

Have you looked at https://zfs.rent/ for cloud storage? You have to provide your own drive, but the monthly cost is relatively low.

You assert that ECC RAM being necessary for ZFS is just a myth but provide no justification for why that is untrue.

Is it not the case that if you don't have ECC memory, ZFS could end up writing a checksum that does not match the data if you get a bitflip in just the right (wrong) spot?

> You assert that ECC RAM being necessary for ZFS is just a myth but provide no justification for why that is untrue.

ZFS without ECC is no more risky than any other file system / software RAID without ECC.

As no one owes you an explanation, it would take you five seconds to Google this and discover:

1. It's been disproven, with one of the original ZFS developers chiming in.

2. The original source of the rumor was a forum post that somehow became canon.

What an incredibly rude comment

> five seconds to Google this and discover

...(after much longer) that it's a rabbit hole with nuance. https://news.ycombinator.com/item?id=18481910

Yes, indeed. ECC RAM is better than non-ECC RAM, also for ZFS.

The myth, popularized by a notorious thread on the TrueNAS forums [1], is specifically that ZFS requires ECC RAM, and will do worse than other filesystems without it, because scrubbing will multiply a single bitflip into a failed pool.

A ZFS core developer says that that isn't the case [2]. Here's some more reasoning [3], also about many other myths.

[1] https://www.truenas.com/community/threads/ecc-vs-non-ecc-ram...

[2] https://news.ycombinator.com/item?id=18480016

[3] https://kldload.com/zfs-wiki/myths

Updated link to the post from Matt Ahrens here[1], which is one of the ZFS creators[2] not "just" a core developer.

[1]: https://arstechnica.com/civis/threads/ars-walkthrough-using-...

[2]: https://en.wikipedia.org/wiki/ZFS#2004%E2%80%932010:_Develop...

Usually if you’re using a NAS you don’t want to lose data, ZFS is not significantly more sensitive than everything else.

But everything is actually quite sensitive.

We’ve accepted lack of ECC because Intel decided it would be a product line differentiator, and serious customers who didn’t want random crashes or to lose data would buy chips with ECC.

It’s actually less of an issue these days because DDR5 has (by spec) some in-line ECC; won’t help with multi-bit errors but its an improvement on what came before.

I know that LPDDR5 has ECC, and not just single bit AFAIK, but if you enable it you lose some memory capacity (the 128 bit bus minus 16 bit for ECC error correction, making it effectively 112 bits)

DDR5 on-die ECC is to achieve acceptable yields in the face of denser process nodes that decrease the reliability of RAM cells. It’s not clear how much of an improvement that is to what we had before, other than allowing for higher RAM speeds. It doesn’t replace side-band ECC.

> We’ve accepted lack of ECC because Intel decided it would be a product line differentiator, and serious customers who didn’t want random crashes or to lose data would buy chips with ECC.

AMD has been allowing ECC on lots of regular hardware for a long time.

People don't tend to buy ECC for desktop use because it costs significantly more (used server ram is/was often cheap... but it often doesn't work in desktop boards), and the performance specs are poor.

My home servers are mostly retired desktops, so they get my old desktop ram and I don't want to pay premium prices for jedec speed ecc ram on my desktops, thanks.

Since DDR5 doesn't include reporting on bit errors (afaik), it likely means much fewer single bit errors, but most experienced errors will be multi-bit. Although, I dunno what proportion of bit errors is on the ram chips and what's on the bus... there's no protection from bus errors.

If there were reporting, you could replace chips with high error rates, but without reporting you'll keep running them until they fail enough to notice.

> AMD has been allowing ECC on lots of regular hardware for a long time.

Often unsupported or untested by the motherboard manufacturer, because the precedent set quite a while ago has stood so strongly.

> because it costs significantly more [...] and the performance specs are poor

Because it's niche. If a large share of the desktop market was using ECC memory, the extra cost would have been 10-12%, and a typical kit would push the clocks and timings at least as much as non-ECC memory.

> Since DDR5 doesn't include reporting on bit errors (afaik), it likely means much fewer single bit errors, but most experienced errors will be multi-bit. Although, I dunno what proportion of bit errors is on the ram chips and what's on the bus... there's no protection from bus errors.

Yeah that sucks. I'm really hoping that DDR6 standardizes a light form of end to end ECC. (Bursts are 24 bits wide and 24 bits deep, so on top of the 512 bits of payload there's a bunch of extra bits that can be used for ECC. If memory chips would just store 4 extra bits per 128, that's enough for reasonable ECC on a per-burst basis.)

> the precedent set quite a while ago has stood so strongly.

I wasn't really around for it, but I appreciate the early penny pinchers that developed fake parity ram. The IBM PC required a parity bit per byte?, but you could add an xor chip to calculate it at read and save out on the expensive ram chip.

I wouldn't mind paying a little more for ECC, what with the extra chip and a little more circuitry, but desktop ECC often starts at 50% more, which I'm way too cheap to pay for.

My favorite parity trick was how the N64 was designed around ECC RAM chips but didn't bother using ECC, so it has a ninth bit at every address that the CPU ignores and the GPU uses as extra storage.

The only hangup with the myth "debunking" is that the point is that the corruption doesn't happen to a per-disk buffer, but to the in-flight data before it's persisted to your stripe.

Which means all copies of the data will be corrupted. This can be anything from an irrecoverable file to complete filesystem corruption.

But generally, yeah, not any more dangerous than any other filesystem, and ECC used to be cheap so it was a no-brainer, you should have backups anyway TBH if we're being honest about storage resiliency.

What has been debunked is the "scrub of death" issue, on a scrub a bad bit flip would cause an error, which would be copied over with good data -- well it was technically good before. It would be statistically difficult to have a fault on a read, then a clean read, then a second bit flip destroying the data.

Had it happen to my PBS backups running on ZFS without ECC just the other day. Turned out to be heat related, memory was fine but during long backups heat was causing bit flips.

I'm also planning a new home server build now and the prices are definitely relatively ass. So far I've spent 820€ on two 22 TB WD Elements HDDs, 375€ on 2x16 GB DDR5 kit, and 520€ on two 2 TB M.2 SSDs (cache). About 1700€ and I still have no server to show for it. Doesn't help that I've been procrastinating on picking the CPU and motherboard.