Puget Systems' Perspective on Intel CPU Instability Issues
pugetsystems.com
pugetsystems.com
That's for detected/known failures: what about random, unable-to-reproduce, hardly noticed the data skip failures?
Have I been living in a fantasy bubble where CPU's do exactly what you asked of them (and errors come from not holding it right)?
Apparently it was much better in 2018: https://www.pugetsystems.com/labs/articles/most-reliable-pc-...
On the other hand, 2011 did show 1.5%: https://www.pugetsystems.com/labs/articles/most-reliable-pc-...
GPU failure rates also weren't great 10-15 years ago, in particular for AMD: https://www.pugetsystems.com/labs/articles/video-card-failur...
It seems high to me too. I can't recall ever having a CPU fail and I must have used/owned hundreds of them in my lifetime. But presumably, failure rates in data centres where CPUs are run 24/7 at high temperatures etc are higher than in consumer applications?
Interesting that failure rates seem to peak in the summer months, too, and this didn't seem to be explained in the article. Perhaps the data center's cooling is working less effectively in summer?
On a 20-sided die a natural one only comes up 5% of the time.
I don't think I'm quite at having owned 40 (desktop) CPUs, but I've had failed CPUs a number of times. On an old Athlon XP 3000+ it had bad CPU cache and was able to change it from crashing repeatedly to working by disabling a large swath of the L2 cache in a bio setting. I am just now retiring a workstation that had an AMD Ryzen 5950x that was generally unstable, that would pass memtest and all the diagnostics I know how to run would pass, but about once a month would print a message on all the consoles I had open about an MCE kernel exception detected on some CPU number at random. One time I had one of those budget TriCore CPUs that had the 4th core turned on in the BIOS by accident, and that generally caused the ton of issues when I figured out it was being detected as a quad core I went in and disabled that last core and it went back to working just fine.
I'm sure at least one or two of my Intel machines failed similarly, I've had a number of dead CPUs including a few Pentium 4s back when I used to fool around with overclocking settings, and at least a few dead server chips. And I know that I've had a few systems that simply wouldn't boot even when all the parts were fresh out of the box but would when I'd swap out one part or another, and sometimes that failed part was a CPU whether it be AMD or intel.
Oh, I also get to cheat no clue how many dead Solaris CPUs I have seen! Those big Mainframe imitation systems have the ability to hot swap CPUs and it was pretty rare to have a cabinet sized computer where every piece of hardware was fully operational, at least in my role as a legacy maintainer of such things. I bet they worked much better when they rolled off the factory line.
My current laptop was getting uncomfortably hot when some random browser pages started pressing the cpu, after searching I noticed that the default setting was to enable some kind of "Boost Mode" (that's basically overclocking in the classic sense), disabling that made a world of difference and looking at the failure rates of the Ryzen 5000 series in the article I'm not a single bit surprised about it.
Googling the laptop family you get tons of Reddit hits, https://www.reddit.com/r/ZephyrusG14/comments/gho535/importa...
I was just trying to provide a few examples of real first hand failures. And most OCing doesn't break anything, but every once in a while you set some voltage and one part never works again, hard not to conclude it was the OC when the failure perfectly coincides. I suppose it could be coincidental, but that stretches credulity.
Another issue as an end user is I don't have the resources to prove that it was the CPU a lot of the time. I've had some laptop failures that could have been the CPU, could have been the motherboard, could have been the power supply, dunno, all I've really got is that it doesn't boot and I don't have the capacity to diagnose it due to the level of integration and inability to get replacement parts to even try to diagnose the problem. And while as a poor college student, I had the time and desire to carefully replace parts and exert maximum diagnostic effort to figure out exactly what is wrong because I can't just buy a new complete setup, not everyone goes to this level of effort, and they may not be correct.
End users really can't pick up on these trends. The data set is so noisy. Sure, the "end users" may have had suspicions about this but I've also seen communities come to consensus about certain things being broken that I had very good reason to believe were wrong and were just internet forums amplifying random loud voices being confidently and loudly wrong about things until it become "common consensus" through the power of nothing but the confident and loud error.
The issue wasn't easy to reproduce, all standard checks and torture tests ran just fine but much more random workload crashed the unit when all cores were maxed for few seconds at a time instead (eg. during compiling). Sometimes it happened twice a day sometimes once a week.
It would only attempt to POST 1 time out of about 10 power on attempts and won't finish enough of the POST to even make it into the BIOS/UEFI setup. When it would fail to even attempt to POST it wouldn't even initialize the USB peripherals (keyboard doesn't light up).
I swapped everything (GPU, ram, power supply, etc) before getting to the point of suspecting the motherboard was faulty because it would not Q-FLASH via a USB thumb drive so I sent it in for warranty service. Since the manufacturer said the motherboard passed their quality checks there is nothing else that could be left besides the CPU.
Even given all that, I refused to believe the CPU could be the culprit until I saw the failure rate graph for AMD 5000 series processors in the article that far exceed what I figure would have been in the 0.5% range.
Live and learn I guess but it certainly adds a new fun dimension to troubleshooting because I don't exactly keep spare CPUs laying around "just in case" like I do with a spare power supply.
For me it was simple: disable C states - Stable for a week (at first it'd be weeks between crashes, then days, then daily, and at this point a couple hours) Enabled them again - crashed within an hour Disable again - Stable for a week. Flop back and fourth a handful of times in the same day.... same as above.
I can only guess that this might be something that was fixed in the mysterious "frequency voltage curve change" stepping.
It took most of a year to nail that one down.
AMD didn't ask for anything besides proof of ownership once I told them disabling C states fixed it. And, 5950x's have a 5 year warranty...
PS: if they were Northwood P4's, see: https://www.overclockers.com/forums/threads/the-official-sud... (I didn't name it. I agree, terribly insensitive name)
How do you know? I thought one of the main ways these were failing resulted blue screens. I'm sure you've had a bunch of blue screens in your lifetime.
If you ever run “cat /proc/cpuinfo” on, say, Skylake - Linux will happily tell you it has 5-6 workarounds active for hardware mistakes.
CPUs are still pretty darn reliable. Think about how many GHz your CPU runs at, multiplied many instructions per cycle there are, and then calculate the failure rate if there was just 1 mistake per minute. Nothing on earth would compare.
> apic_c1e spectre_v1 spectre_v2 spec_store_bypass swapgs taa itlb_multihit srbds mmio_stale_data retbleed eibrs_pbrsb gds bhi
> apic_c1e
Both ACPI and MP specifications require that the APIC id in the respective tables must be the same as the APIC id in CPUID.
The kernel retrieves the physical package id from the APIC id during the ACPI/MP table scan and builds the physical to logical package map.
There exist Virtualbox and Xen implementations which violate the spec. As a result the physical to logical package map, which relies on the ACPI/MP tables does not work on those systems, because the CPUID initialized physical package id does not match the firmware id. This causes system crashes and malfunction due to invalid package mappings.
The only way to cure this is to sanitize the physical package id after the CPUID enumeration and yell when the APIC ids are different. If the physical package IDs differ use the package information from the ACPI/MP tables so the existing logical package map just works.
> taa
TAA is a hardware vulnerability that allows unprivileged speculative access to data which is available in various CPU internal buffers by using asynchronous aborts within an Intel TSX transactional region.
> itlb_multihit
iTLB multihit is an erratum where some processors may incur a machine check error, possibly resulting in an unrecoverable CPU lockup, when an instruction fetch hits multiple entries in the instruction TLB. This can occur when the page size is changed along with either the physical address or cache type. A malicious guest running on a virtualized system can exploit this erratum to perform a denial of service attack.
> srbds
SRBDS is a hardware vulnerability that allows MDS techniques to infer values returned from special register accesses. Special register accesses are accesses to off core registers. According to Intel's evaluation, the special register reads that have a security expectation of privacy are RDRAND, RDSEED and SGX EGETKEY.
> mmio_stale_data
Processor MMIO Stale Data Vulnerabilities are a class of memory-mapped I/O (MMIO) vulnerabilities that can expose data. The sequences of operations for exposing data range from simple to very complex. Because most of the vulnerabilities require the attacker to have access to MMIO, many environments are not affected. System environments using virtualization where MMIO access is provided to untrusted guests may need mitigation. These vulnerabilities are not transient execution attacks. However, these vulnerabilities may propagate stale data into core fill buffers where the data can subsequently be inferred by an unmitigated transient execution attack. Mitigation for these vulnerabilities includes a combination of microcode update and software changes, depending on the platform and usage model. Some of these mitigations are similar to those used to mitigate Microarchitectural Data Sampling (MDS) or those used to mitigate Special Register Buffer Data Sampling (SRBDS).
You can argue that the system shouldn't crash (although at that low a level, not sure what else can happen)...
but beyond that, how is "VirtualBox and Xen implementations violate the spec" a failing of a CPU?
(I’m the one who characterized this issue on Linux and wrote the test case and the workaround.)
I guess it would be a pretty useless comparison if they weren't carefully filtering for CPU-only failure though..
Seen other HW issues, memory or motherboard(soldered memory) on my ex's laptop that affected only certain adress ranges, memtest86 has been my go-to to check computer health when random crap starts happening since then and I've replaced at least one memory stick on another machine thanks to it.
These are gaming CPUs clocked right up to the threshold of stability (and in some cases, past it)
Server CPUs with ECC RAM are significantly better.
However, if you haven’t experienced much CPU instability, you may not have operated at scale where it appears. Get into the scale where operations occur across 100,000s of CPUs and weird things happen constantly.
Most Raptor Lake "server boards" right now are W680 with client CPUs because the C266/Xeon E-2400 took a long time to come out. The one intended for workstations typically has overclockable settings or is even overclocked by default, which means it's likely to get hit with the failure (1). The one intended for servers do have more conservative settings, but can still be hit with failure (2) under some conditions.
Buildzoid released a video on the Supermicro W680 blade a bit ago that were having issues after running a single-core load 24x7, which is essentially 24x7 boost[1] (aka issue (2)). Xeon E-2400 _could_ be affected in this scenario, although even the highest clock E-2400 SKU (E-2488) is only running at 5.6GHz without Thermal Velocity Boost, and most others are ranging from 4.5 to 5.2 GHz boost (rather than the 5.8 to 6 GHz boost some client SKUs do). I feel like the actual B0 Xeon E-2400 would be a lot less prone to both failures (1) and (2) due to this (but it could happen, though there's no reports of such).
But then the conversation gets muddied enough that "even servers and Xeons are affected" becomes the common narrative (while the former is true, the circumstances needs to be noted; and for Xeons, it's a _maybe_ at most, since right now there's no report of Xeon E-2400 failing).
All the evidence I've seen points to electromigration as a cause of this degradation, and IMHO excessively aggressive automatic overvolting by Intel's microcode is to blame.
There is actually a simple experiment which can determine whether that is true --- remove the fan from the heatsink, or even let the CPU run without a heatsink. As the CPU will automatically throttle once it reaches its designed maximum temperature (and AFAIK that is a hardcoded limit), it will lower its frequency and voltage to maintain that temperature. If this results in a stable CPU, while the one that has great cooling becomes more unstable, it confirms the hypothesis.
There are numerous stories of machines where the heatsink was not in contact with the CPU for some reason, yet they remained perfectly stable (but slow) for many years. I can also say that I've had an 8th-gen i7 running at 100% 24x7 with all power and turbo limits disabled, with its temperature constantly at the design limit of 100C, and it has also remained stable for over 5 years.
I once had a laptop, which came from the factory with the four screws which hold the heatsink to the CPU missing. It was very slow, and shut down after a few minutes (the reason being thermal shutdown in the BIOS event log helped diagnose the issue). After the four screws were replaced (each screw came in its own large individual cardboard box), it worked fine for many years, BUT after a couple of years (still under warranty), the motherboard failed with a short in the power input. I suspect that all the extra heat from when the CPU was without a working heatsink went to the power supply components through the motherboard ground plane, and cooked them, significantly shortening their useful life.
Turbo Boost (and Thermal Velocity Boost if applicable) frequencies are according to specifications, it's not an overclock.
Download OCCT. https://www.ocbase.com/
disable xmp
stress test with OCCT
if it crashes here, you have to downclock your cpu
enable xmp
stress test with OCCT
if it crashes here you need to downclock your memory speed
repeat 5 and 6 in the smallest interval of ram speeds in your bios until it's stable for ~5m
Now, I'm sharing this here with because it's good to know but also to make the point: do you think supermicro is doing this for every server that leaves their factory? Not to say there isn't an issue with Intel, but based on what the article says about failure rates, and what I see from friends with AMD, there is an issue with system stability in general that extends beyond this specific issue. My guess BIOS settings being ran at redline didn't do well when we rolled out the new DDR5.
Both Intel and AMD publish their memory controller specs and you should thoroughly understand them if you do want to overclock (read: use XMP/EXPO), anything that goes above the specs is not guaranteed to work.
Incidentally, for all the flak the motherboard vendors rightfully got with their out of the box overclocking defaults, their default configuration for RAM is in fact to stick to Intel/AMD specifications like superglue.
Overclocks are explicitly not covered by warranty.
Yes, Intel and AMD and mobo vendors all say overclocks "may" void warranty and in general they have honored warranties for overclocked hardware, but the official and legal position is that overclocking is not covered by warranty.
>It just doesn't fit the facts to say that you shouldn't expect 5400 Mhz ram to work at 5400 Mhz
You can certainly try RMAing the RAM with the RAM vendor since they sold it for whatever frequency they marketed it at. But as far as Intel and the mobo vendor would be concerned, an overclock is beyond the purview of their warranties.
Puget's numbers kind of vindicate that by showing 11th gen was even worse and AMD clearly benefitted from their "underdog", "cult favorite" status.
It would be nice if we could be rid of most of this noise so we could get down to what truly matters.
Unless you do a very thorough timeline, you might be confused. But go dig into the last 6 months and you'll see that Intel has either no idea or are absolutely muddying the waters. Neither of these cognitive conclusions should result in Intel looking anything but half rate purveyors of silicon.
Obviously Intel messed up badly, and those settings shouldn’t result in this behavior, but maybe this will convince system integrators to have more reasonable defaults in the future.
In a top end system, we’re already sitting in territory where our GPU is our benchmark, do we really need to default to giving the cpu so much power?
The defects of Zen 4 are random manufacturing defects, so most of them are detected by Puget after assembling and testing their systems, before selling them to customers.
On the other hand, most of the Raptor Lake defects happen after some time after selling them to the customers, which implies some kind of wearing mechanism, which either can affect any Raptor Lake CPU or perhaps only CPUs that have some kind of latent defects.
Because the Raptor Lake defects happen after some time, it is likely that their number will continue to raise among the already sold systems and the same statistics recomputed after some months might show a higher number of Raptor Lake defects than now.
Intel trying to pass the buck is as much of a problem as the CPU's themselves really, because now you can't trust them.
Intel programming the voltage curves wrong is on them, but that doesn't matter if the motherboards aren't going to run the CPUs according to specification out of the box. Intel calling out mobo vendors for their stupid defaults was justified and very much needed.
Incidentally, there is no such thing as a "baseline". Intel separately specifies an "Extreme Config" for applicable SKUs (the i9s), but otherwise there is only the one set of specifications.
The fact you are talking about "baseline" suggests you did not actually consult the specifications published by Intel, just like the mobo vendors who put out so-called "Intel Baseline Profiles" before they got chastised again for not actually reading and obeying the specs (and arguably they still don't).
[1]: https://edc.intel.com/content/www/us/en/design/products/plat...
[2]: https://edc.intel.com/content/www/us/en/design/products/plat...
[1]https://community.intel.com/t5/Processors/June-2024-Guidance...
Yeah, you still haven't read the specifications.
Please read the fucking specifications if you are going to partake in discussions concerning specifications.
Extreme is "incomplete" because those specifications apply and only apply to Raptor Lake i9 SKUs. "Baseline" is incomplete and not recommended because "baseline" does not exist in the specifications.
What's more, "performance" also does not exist in the specifications per se. Most of it is actually the specifications copied verbatim, except for PL1 which is 125W for the concerned SKUs according to specification and actually noted as such by Intel in that chart.
The chart also excludes other important information, such as the PL2 time limits (56 seconds for the SKUs in the chart), the max core voltage rating of 1.72V, and AC/DC load lines and associated calibration.
Again: Please read the fucking specifications. You are contributing to the media sensationalism and emotional chest thumping, which is all worthless noise.
Have I misunderstood the graph or is it actually that awful?
> Even though failure rates (as a percentage) are the most consequential, I think showing the absolute number of failures illustrates our experience best
> Everything I’ve shown you so far is our raw number of failures, but what matters most is failure rate percentages. Let’s look at total failure rates in the context of multiple generations and with comparison to AMD Ryzen CPUs.
The Costco desktops in stock at the local Costco are all 13th/14th gen systems priced reasonably enough with Nvidia 4060 or 4070 cards... but my perception is they are not selling well because of the concerns raised about these CPUs.
We've been on 15th gen (Meteor Lake) for quite a while now, with 16th gen (Arrow and Lunar Lakes) presumably coming later this year. I sympathize with Costco wanting to move all that old stock out now rather than later.
No, I am not going to entertain the desktop/mobile generation split bullshit that AMD also has. Screw that noise, disjointed market segmentation is anti-consumer.
(No, I am not going to entertain Intel's new Core XX branding either.)
Going by the more common theme of "generation goes up by 1 each year", Arrow lake would been 15th gen.
Meteor lake came out the same year as 14th gen, but the "14" branded parts are raptor lake refresh instead of meteor lake due to the deadline miss, they couldn't get the new architecture ported to all product segments so they just did consumer laptops since they don't pull much power and don't push the silicon too hard.
The only real problem with AMD is that they are gouging on APUs. Who exactly is in the market for a $330 (from memory) APU for their office PC or gaming PC? Of course, they already dropped the price, but that $330 launch price felt really awkward.
The 185H followed the 1365U, so the 185H would've otherwise been the 1465U.
It was supposed to also be the 14th gen of desktop parts but they couldn't finish it in time, so we got raptor lake refresh instead. There is still the line of e.g. 14700HX (Q1 24), which is a raptor lake laptop part that came out after meteor lake (Core Ultra 9 185H) did (Q4 23), but it beats meteor lake by being used in 45W+ CPU TDP gaming laptops paired with a discrete GPU.
Not one vendor in 100 would stand up and say something like that in public. The system I bought from them has been great, and if/when that changes, I don't doubt that they'll have my back when it comes to haggling with Intel.
If money is no object -- or developer time is expensive -- or failure is expensive -- it'd be my go to source.
For what I'm doing now (education), pricing is quite a bit too high.
That last little bit is a thing that has infuriated me for decades, and it's all over the place in anything that tech touches, not just motherboards. Can't we call things by their proper names?
CPU and GPU learned from decades of RAM business.
Sell garbage. People buy, get blue screen, blame software. ECC? pffft.
Let's just see how long it will take for NVMEs to fall to the same level of reliability as usb pen drives and everyone will just buy 4 of them to RAID1 and assume that's just "normal".
> We have enough data to know that we don’t have an acute problem on the horizon with 13th Gen — it is more of a slow burn.