"Some workstation users will need ECC memory, and up to 512GB of it. When memory has an error rate of 1 error per GB per year, using 512GB ensures almost two bit errors per day: something that a 60-day simulation would find catastrophic."
In truth, there are plenty of good reasons, both for data integrity and security, to have ECC. However, there are real costs to ECC too - 10-30% immediate performance overhead (depending on your application) and much more than that for memory-bound workloads (due to lower frequencies and subtiming issues), not to mention hardware cost efficiency. Instead of oscillating between "ECC for everything" and "ECC is useless" people should rationally compare the costs and benefits for their specific workload and choose accordingly.
In terms of speed it's ~2–3 percent slower worst case, and memory is only one component of overall performance lowering the impact further.
More importantly, bit errors are not evenly distributed with some users seeing vastly more errors.
I am one of the principle engineers working on a game that was able to support over 1.2 million players concurrently, so I feel that I have some merit behind what I'm about to say.
ECC Memory does not have such an extreme overhead. Our gameserver is memory bandwidth bound. To put that into perspective a change from DDR3 to DDR4 saw a jump in performance of 34%~. Using non-ecc memory (which is obviously very available to the development team) only gave a penalty of roughly 3% in the worst case. (both DDR4 systems).
Why are we memory bandwidth bound? because we allocate 250GB of "game world" and phases and make teeny tiny modifications to them when you run around or pick up loot and shit. It's as taxing as you can be on memory bandwidth since you can't batch set or load anything.
On a further note, while the implementation of ECC itself is very fast for modern RAM, the same is not true of VRAM on GPUs. At least as implemented in NVidia Tesla GPUs, enabling ECC has a huge performance penalty by itself. [1]
[1] http://geoceano.fr/wp-content/uploads/2015/01/cpu-gpu-lbm-in...
When you start looking at multi CPU computers with 1+TB of ram the story changes.
PS: Some of this stuff is also really hard to test. Without ECC it's hard to data in terms of stability for petabytes of ram over years. And no extrapolations don't work very well.
Now, non ECC memory might have better specs / a higher overclock rate etc, but doing either of those increases the rate of errors. And sure you might not notice the problem directly, but that does not mean things are ok.
The value of ECC is often more to detect errors than to correct them. Remember, if you have 10 errors an hour then you unlikely to see them in a 20 second memory test.
In practice out of 10,000 machines you might have 1/2 your errors on 10 of them. Critically, if you have ECC memory and you both detect and replace those 10 machines the apparent rate of failures then drops.
PS: This is a little dated (09) and only from a single set. But, still an interesting introduction: https://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf "Around 20% of DIMMs in Platform A and B are affected by correctable errors per year, compared to less than 4% of DIMMs in Platform C and D. ... About a third of all machines in the fleet experience at least one memory error per year (see column CE Incid. %) and the average number of correctable errors per year is over 22,000. "
EDIT: the least out of date reference I could find is a blog post by Jeff Atwood, where he concludes that based on the available literature ECC-correctable errors are quite rare. [1]
"The median number of errors per year for those machines that experience at least one error ranges from 25 to 611." "We find that for all platforms, 20% of the machines with errors make up more than 90% of all observed errors for that platform."
A minority of that 20% again have even more issues.
The worse case is literally 10,000x more than 1 bit error per year on a single machine. Without using ECC you really can't tell other than some vague not sure what's wrong just replace it.
Hmm, like what? I've run many simulations too and had the opposite experience. It's hard to implement checks because your algorithm defines how the simulation should permute over time.
While a bit error could certainly crash the simulation, long running simulations can be resumed from checkpoints with minimal loss of time and the probability of a crash is still very low for most real workloads.
Not really. Your algorithm determines how the simulation plays out, so you can't really notice small errors until much later. By then it's hard to roll back to an earlier state unless you're saving 64GB snapshots.
[*] There are exceptions in specific circumstances most of which would produce obviously wrong results or crash the simulation outright. However, I'm not claiming that single bitflips are never an issue for simulations, only that it happens much less often than people sometimes think.
Also, it's not just physics simulations that are at risk. Voxel renderings are especially susceptible to one-bit errors.
Anyway, I haven't argued against using ECC memory in any of these comments. I've argued that you should know your needs and choose accordingly, and that ECC memory isn't free and therefore cannot be a universally optimal default for everyone.
For much of my physics work I've found ECC memory not to be worth the cost at all. For other things, like most database servers (and perhaps voxel rendering, I don't have enough experience there to comment) ECC memory is clearly worth it. I just think people should invest some thought into the question before evangelizing for or against ECC in the general case.
It’s some kind of snobbery to advocate that good technology should be held back until non-enterprise plebs demonstrate that they are deserving of it.
Just give everyone the good technology as much as is possible. Who’s it hurting if some subjectively underutilise it?
Everyone who buys the product and doesn't require the technology, because they'll be spending more money on something they don't need or use.
It's a political decision by Intel to make ECC an Xeon-only product which has resulted in the vast majority of the price difference. The silicon die area required for it really doesn't cost that much more - often, the area is present on the consumer dies but just disabled so they can charge a lot more to the enterprise customers who actually require it. And political decisions like that make a lot of HN users unhappy.
As Joel Spolsky said in https://www.joelonsoftware.com/2004/12/15/camels-and-rubber-...:
> Working my way backwards, this business about segmenting? It pisses the heck off of people. People want to feel they’re paying a fair price.... And God help you if an A-list blogger finds out that your premium printer is identical to the cheap printer, with the speed inhibitor turned off.
Everyone knows that ECC parts are nearly identical to the cheap parts, with the ECC module turned off. But there's no boycott, because there are (were?) no other options, like there might be for a printer company. ECC segmentation is just something everyone knows and resents.
Computer reliability should be a rising tide. Up-and-to-the-right is the goal.
Scientists, Engineers and Economists shouldn't use a Phone or MacBook? Your handle is ironic.
The lack of ECC is a method to artificially segment markets and nothing more.
Not ECC, but "single bit errors" can be fatal in other domains [0]
[0] https://gizmodo.com/382026/a-cellphones-missing-dot-kills-tw...
"During the logging period there were a total of 52,317 bitsquat requests from 12,949 unique IP addresses. When not counting 3 events that caused extraordinary amounts of traffic, an average of 59 unique IPs per day made HTTP requests to my 32 bitsquat domains."
http://dinaburg.org/bitsquatting.html
Overheating can be a cause of bit errors, and heat is a concern in mobile devices and laptops.
That's a misconception, to quote Matthew Ahrens, cofounder of ZFS: "There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem."
See also [0].
[0] http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-yo...
https://research.cs.wisc.edu/wind/Publications/zfs-corruptio...
ZFS is a lot more active... checksumming and caching mean that important information is spending time in RAM. It's not good if that information gets corrupted.
ZFS doesn't cache in any special way, or more than your typical filesystem.
Lack of ECC just lessens the benefits of ZFS, it doesn't exacerbate any problems.
It's not that ZFS without ECC is completely unsafe... it's just not as safe as it could be. ECC RAM becomes the next biggest concern once you've addressed on-disk bitrot.
I was hinting that you still get a bit of extra protection with ZFS over other filesystems, both on Non-ECC RAM. There is a chance with a stuck/corrupt bit that checksum will fail and you will get a read error. I interpreted OP as saying ECC is a must for ZFS, but I don't think you are more prone to corruption that any other FS?
I'm basing my understanding on this: http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-yo...
These checksums are fixed exactly once, in the zio pipeline, as a dirtied buffer is being passed through it during a write operation. The checksum then stays on disk in that form in block B for as long as block A is reachable.
In the read case, checksum validation is done at the vdev_{raidz,mirror,...}.c level and repaired there if possible e.g. from another element of the mirror; recovery is also possible in the case of copies=N (N > 1). Generally, the checksums of read data and metadata are not used once a buffer is successfully in the ARC. (However, compare arc_buf_{freeze,thaw}(), which one can enable, and which will happily cause panics if ARC buffers are corrupted whether through physical problems (e.g. in memory, in a bus, in the CPU) or software error in the kernel.)
Block B may already be in cache for some other reason, in which case the in-memory checksum is used (this is the "yes" part of "yes-and-no"); otherwise, block B will have to be fetched before block A, because block B is where A's checksum lives. Block B may be evicted from ARC long before block A is evicted; any read() or similar call that needs data from A will not result in B being read back into memory.
A buffer kept in ARC and used as the source of subsequently-dirtied data will have its checksum generated anew, just like the fresh write above. Obviously if the data it is dirtied with is bad, it will go to disk and its parents will have a checksum reflecting the bad data. However, the source of the badness need not be within the transient dirtied buffer, nor in a long-resident ARC buffer; userland memory can be corrupted too.
If one has bad memory than bad data may end up on disk, checksummed in such a way that the badness will not be detected by the ZFS subsystem. But this is really no different from any other RAID-like data-validating system or a traditional one-disk filesystem which doesn't checksum. If bits are flipped in userland data that is ultimately subjected to a write(2) call (etc., think mmap()), no filesystem can reasonably expect to do anything other than write out the data handed to it via the system call.
Like any other filesystem, ZFS has its own metadata for tracking allocations, object metadata, and in generating the branch-to-root Merkle tree. The wrong memory corruption (and this has happened in software during ZFS development) can result in data loss or even a pool too damaged to be imported (go to backups). ZFS is not especially more exposed to that than any other filesystem.
Since the arrival of compressed ARC and crypto, only a small fraction of ARC buffers are kept uncompressed or unencrypted in memory at any time (this is controlled by the dbuf cache tunables). Bitflips in any encrypted-in-memory ARC buffer will be caught (noisily) if the buffer is subsequently read or modified, since a decryption will be done at that time and would fail under a single modified bit. Many possible corruptions in compressed-in-memory data will also be caught depending on the type of compression used when the buffer was first written out to disk. In neither case is the ZFS checksum involved in the catching of such corruptions.
Finally, this still won't protect against corruptions in userland, corruptions which hit the various data structure that point to buffers in ARC, or corruptions in the text segments in the zfs subsystem or elsewhere in the kernel. However, ZFS is not realisically more fragile to such corruptions than any other filesystem.
What ZFS really needs is native encryption to be generally available.
Does anyone have an idea of the cost increase for using ECC RAM in a mobile device for instance? Are there also power usage concerns to worry about?
AFAIK there are basically zero consumer devices using ECC RAM.
I want to say I had ECC in my old AMD K6-350, but that was a lot of years ago. Either way, I'm pretty sure they have supported it for a long time and in many of their CPUs, even the CPUs aimed at consumers.
Here's a comment I made last night, if you want to dig in and find some current offerings:
Same reason ECC is not supported on consumer processors from Intel. Here it would be almost trivial to add, but they don't.
Simple business strategy.
So, they're okay with it on consumer parts, as long as you're not looking to do anything resembling professional work I guess.
Technically. A fraction of a percent increase in die size for the memory transceivers, basically nothing for making and verifying checksums.
It's all business strategy.
manufacturing cost is non-linear. Obviously intrinsically word gunk.
And in a lucky coincidence it turns out that those who want ECC are willing to pay the moon for it, which of course gatekeepers like Intel exploit.
Basically there's no evidence that ECC memory is signficantly slower or more expensive than non-ECC memory. Obviously it's slightly more complex than non-ECC memory, but it's not a particularly high-tech addition. For all its supposed complexity/slowness, Samsung is using it in its caches on their Exynos SoCs.
Everything can be perfectly explained in terms of market segmentation.
However, in many applications we find that using (forward) error correction almost always increases data density (for storage) or bandwidth (for transmission), simply because a FEC stream does not require a nearly-perfect channel any more. This is the way hard disks, SSDs, WiFi, LTE, DSL, ..., satellite communications, ...[, ...][, ...] are able to cram incredible amounts of data into very noisy channels. Thus, ECC significantly lowers cost in many dimensions (be it frequency spectra, storage prices, not having to re-cable entire countries...).
(And if you don't use the extra noise margin to increase density/bandwidth, then you can use it to increase reliability, like we usually do with ECC memory)
Thinking about it for a few minutes, the memory bus will most likely be the only bus in your computer that has no error correction/detection. USB, SATA, PCIe, all of them require it. The main memory will also most likely be the only storage that doesn't use it (apart from firmware flash chips and the like, but these often use a checksum at least).
Do you mean that ECC has those benefits, or that other applications of error correcting codes has them?
Personally, I've always found this to be very annoying. The only people using non-ECC memory should be Xtreme Gamer type people who don't care about their data and only care about squeezing that last 2% out of their system to land at the top of the penis size chart. Non-ECC should be a feature like liquid N2 cooling.
The moral of this story is that ECC is only ever advertised explicitly, when it is special to have, i.e. because there are segments in the market where it is not used.
(A somewhat curious case are small microcontrollers. Often these control relatively important aspects of machinery, for example, drives [say a bit-flip in the drive controller turns STOP into RUN, which can get rather ugly on e.g. a lathe]; these usually don't have ECC. However, they are manufactured in very coarse processes and generally don't use DRAM.)
The real question though for a large scale consumer product manufacturer is how much margin they could convince the RAM makers to give up on ECC. Once it became standard, it probably would not cost much more than non ECC. Maybe 15% more? (1/8th extra memory, plus some extra for other expenses.)
It is nice to see "EDAC amd64: DRAM ECC enabled." in my dmesg output. :) That, and the NVMe SSD is screaming fast.
I'm debating whether to try my hand at overclocking my R7-1700, but even leaving it at the stock clocking it's pretty speedy, and it runs cool with the stock cooler.
See e.g. this paper by Walker and Betz from the San Diego Supercomputing Centre (paper from XSEDE '13) on the effect of ECC on GPUs for MD simulations: https://dl.acm.org/citation.cfm?id=2484774
Even if it's just in a float, it looks like a single bitflip could change FLOAT_MAX to NaN. That could hurt.
You can also use ptrace to corrupt the memory of a process more subtly.