Hacker News Reader

Statistically, in my experience, plain old memory corruption bugs within your code or within the GPU itself are a more likely explanation than issues ECC would fix.

Games have pretty terrible quality because games can remain playable in the face of bugs. GPUs have similar issues because numerical accuracy is an after thought but numerical accuracy issues can cause issues in code that assume stability of results.

Things have gotten better particular on the GPU vendor side of things, but still, memory safety issues and UB issues in c/C++ code far dominate as the root cause any issue you’re likely to see vs memory bit flips (unless people are running very overclocked and unstable machines).

I'm very much aware of issues mistakes around memory can cause, and we use many tools (including different languages) to avoid them or verify code. My point is even taking that into account there's a long tail of unexplained issues.

And UB shouldn't be a scary thing 99% of the time, as you won't be hitting it anyway (or its actually the result of another real bug, like not handling overflows etc.), though as at some level you start trying on platform specific behavior and start /defining/ them. And again, there are tools and options around that, or highlighting areas where you might still be relying on them. They just might be toolchain and/or platform specific. It's never been the scary monster some online programming language fanatics seem to make it sound like, but is something to be aware of and managed.

And

> unless people are running very overclocked and unstable machines

Yes. Have you seen the Gamer Hardware Enthusiast community?

> And UB shouldn't be a scary thing 99% of the time, as you won't be hitting it anyway (or its actually the result of another real bug,

This is exactly how exploits are made. Accessing UB usually is the result of "another bug" because the UB isnt a bug itself.

>> unless people are running very overclocked and unstable machines

> Yes. Have you seen the Gamer Hardware Enthusiast community?

Having seen it for the last 2 decades: A lot of them are interested in OC, but nobody is doing it. Maybe 5%, most likely <2%. Also, most modern Motherboards do a "RAM-Training" session before boot. On unstable machines, this will result in a "test failed" once in a while, with the motherboard showing a "Boot failed, returned every clock setting to default" message, which most likely is not being read by the user, who is either not at the PC the moment it happens or in "why is this taking so long? skip, skip, skip"- mode.

What does happen is that Motherboards have OC-like settings out-of-the-box/per default, which were stable most of the time and de facto-accepted by Intel, however is now hitting issues and diminishing returns:

https://www.radgametools.com/oodleintel.htm https://www.igorslab.de/en/intel-spielt-mit-dem-namen-und-de...

With XMP many motherboards effectively overclock ram out of the box.

What I see a lot of too is poor quality hardware causing crashes - eg poor quality power supplies, actually faulty memory, memory that is marginal at the speeds that it's sold at (eg fine at the standard JEDEC speeds, but fails at the XMP speeds), but also poor cooling - eg not enough airflow in the chassis paired with a modern GPU.

I had a PC that was having inexplicable game crashes, but only in stressful games - swapping the power supply from an existing ~10 year old one (which was still rated as enough capacity for my hardware) to a brand new ATX 3.0 unit resolved it.

> Yes. Have you seen the Gamer Hardware Enthusiast community?

I'd venture 95% of people run stock, 4% do undervolting and 1% do overclocking.

Which isn't that strange because GPU vendors have become relatively adept at pushing the silicon near its max. Its why you can run an aggressive undervolt with a few percentage point lower clocks and be rewarded with a 20-30% wattage drop with commensurate drop temperatures and fan noise, all whilst losing perhaps <5% performance.

But the GPU vendors want to put big number on box so they'll have users suffer loud fans :-)

> I'd venture 95% of people run stock, 4% do undervolting and 1% do overclocking.

As others are pointing out, there's an ongoing issue where it seems some products push it beyond the point of stability by default.

I think I'm more pointing out the trend that even "informed" purchasers just look at the benchmark charts. There's no other information required.

Perhaps things like the mentioned instability making the news will change things going forward? But not holding my breath. If there's still advantages to pushing it beyond the limit, then manufacturers will do so.

> And UB shouldn't be a scary thing 99% of the time, as you won't be hitting it anyway (or its actually the result of another real bug, like not handling overflows etc.), though as at some level you start trying on platform specific behavior and start /defining/ them

I’m talking about language-level UB - that’s not anything you can rely on on any toolchain / platform. And UB will also manifest as random unexplainable crashes in random spots just like memory corruption will.

As for tooling, unless you’re using something 100% memory safe with absolutely no call out to unsafe code (which definitely isn’t a game/GPU scenario), you’re going to have this risk & it almost doesn’t matter how much you test or use memory checkers because the long tail of issues are going to be the result of a really difficult to reproduce sequence of events. Additionally, if you have any multi-threaded code, all your testing goes out the window because I’ve seen many concurrency bugs that have hidden in plain sight in very public spots until someone figured out a repro (e.g. https://reviews.llvm.org/D114119, https://probablydance.com/2022/09/17/finding-the-second-bug-...). Race conditions are notoriously difficult to reproduce in controlled environments.

Oh and when I said you’re safe if you’re in a 100% memory safe language? I lied. There are compiler bugs that could be misgenerating your code + you’re likely running unsafe constructs somewhere in your code that has access to your memory space (whether the OS kernel, whatever is making syscalls to the kernel on your behalf, something that eventually calls the C library, something that is needed for performance, etc etc etc) not to mention bugs in the compiler/JIT that either allow unsafe/unsound constructs or just straight up miscompile correct code. And finally, there are sources of HW issues unrelated to memory bitflips (your CPU is super complex and has bugs too you know as does your memory controller).

> research has shown that the majority of one-off soft errors in DRAM chips occur as a result of background radiation, chiefly neutrons from cosmic ray secondaries,

> Recent studies[5] show that single-event upsets due to cosmic radiation have been dropping dramatically with process geometry and previous concerns over increasing bit cell error rates are unfounded.

So aside from the fact that DRAM susceptibility to cosmic rays has been decreasing (I’m not convinced with the Wikipedia explanation - an alternate explanation can be that the percentage of critically important data in DRAM has shrunk as a percentage as the overall capacity has increased), you’re argument would be that random cosmic radiation is going to randomly hit the DRAM cell containing your code / critical data. On the order of things that are likely, that’s the last thing.

Oh and all this relies on you correctly grouping related crashes correctly and I’ve generally seen that to be a significant challenge on any project I’ve participated on (e.g. Oculus for the longest time would group unrelated crash reports and not group the same ones correctly although I helped the team try to make progress on that).

Again, it’s not impossible that it’s a legit HW corruption issue. However, all the engineers I know frequently also blamed cosmic rays but at the end it’s all just shorthand for “not worth wasting time trying to track down because it’s in the tail of issues you’ll never get to“.

Also statistically and in my experience, a low probability event multiplied by a large number of units means millions of people hit hardware failures. It's not at all surprising that someone working on graphics drivers ends up seeing a lot of it.

Yes, and the issues that ECC can correct for are a small fraction of that overall HW failure rate. And SW issues will always still dominate.

There are certain niches where the hardware issues will actually dominate the queue of issues and support requests.

One I can relate to is if you have something with low code churn and extremely logically simple that does many gigabytes of I/O per machine over a short period. The maintainer of that code is going to see tons of bad disks. That was me once.

I forgot the source, but I recall some game dev telling that they had taken to run a basic stability test every time the game launched. Half the crash reports came from systems that failed the test, so those were summarily ignored.

There is an error correction layer in DDR5, could that be spotted statistically as a difference between Ryzen 5000 and 7000 processors?

We did this in Guild Wars, yeah. I'm sure lots of other games are doing it by now. There were a significant number of Problem Customer PCs that would just crash all the time because of stuff that was obviously a CPU or RAM defect, though I don't know if it was half our crash reports.

What kind of stability test is quick enough to run every time the game starts and good enough to detect real problems?

I believe it was actually an ambient check of things like floating point arithmetic that ran on an ongoing basis, and if it ever failed we popped up an error message.

We already shipped with debug assertions enabled in release builds, so it wouldn't have been unprecedented to do things like validate result bytes after doing big memcpy operations or things like that. Not sure if we did, though.

What was the difference between CPU and video card vendors (if you can talk about that at all)?

And then you have car companies like Tesla that removed ECC memory in newer iterations of their car computers to save money.

At worst, you suddenly veer off the road and roll a couple of times, no biggie.

It's not a rocket or something dangerous, right?

Also, newer iterations of the car will be even more cheaper due to cut corners, so you may save some money in the process, too.

I followed a Tesla yesterday. It was clearly on autopilot along this single lane road and it was veering all over the place like a drunk person, sometimes into the verge, sometimes over the double white line (the line you should not cross in the UK) and obviously making micro adjustments to steering with periodic random braking. I don't see how someone could relax as a passenger in that. Shockingly bad.

The latest fsd version definitely doesn't do that, you can look up videos on yt

Perhaps it really was a drunk driver then!

I wonder if the software quality is lower for self-driving in non-US roads. I assume there would be less focus on development and testing in other markets.

I remember reading that Tesla can't recognize lighted or dynamic road signs (which are essentially a matrix of lights which change according to road conditions) present in parts of Europe because they're not present in the US to begin with.

Same signs were easily detected by a Ford Puma I drove recently on a highway.

I assume training data for European roads are also is of inferior quality due to amount of miles traveled and because of sheer variety of the roads, road signs and norms around Europe.

Early in my career I had to deal with similar issues, but with cellular networking equipment. I do think it's one of those things where if you have a way to detect the bit flips it happens way more than you think... but most people can't detect them and mostly get lucky enough that they never see an impact when it does happen.

But because we had some equipment that would checksum itself we were able to see the bit flips happen, probably every 1-3 weeks or so I'd find a bit flip.

I literally just built a new workstation with 256GB of DDR5 ECC memory and had two detected bitflips within the first hour. It's likely that something's off with the timings or voltages, then, but... if I didn't have ECC, didn't have an AMD processor that generates MCEs Linux can decode, or didn't have a kernel set to log MCEs, would I know about them? Definitely not.

I’ve found crashing in games using NVIDIA’s DX12 driver is one of the most reliable indicators of CPU/RAM instability in gaming PCs, so doesn’t surprise me!

Lots of people’s gaming PCs (self built and otherwise) are very crashy - there’s a surprising number of people who get regular BSODs who just accept it as “normal”.

Also see the recent "UE5 is GARBAGE it KEEPS CRASHING on my rock-solid 14900K" episode.

I got ECC on my previous system (at great cost) but had to settle for non-ECC for my current workstation since the cost of Threadripper builds has gone up so much since the 3xxx era. It's really frustrating, even if DDR5 (supposedly) has much lower error rates. Every time a game crashes I'm going to suspect the RAM.

Back when I heavily used GCP's GPUs I'd actually get failed tests with ECC errors. It was kind of neat.