Hacker News Reader

Comment by jandrese | Hacker News Reader

The strangest part of the whole thing is how they claim that the CPUs will work fine initially, but then degrade over time in a deterministic way. How is nobody else seeing this? Are these guys executing code paths that are normally not used? It sounds a bit like their code is buggy in a strange way that doesn't cause problems on AMD but does on later generation Intel, but that wouldn't explain why code would work initially but fail later unless maybe the problem was brought about by a microcode update from Intel?

I fully agree that demanding Intel recall all of the CPUs at this point is premature. They've done nowhere near enough research to pin the problem on the processor and not their code.

View on HN

> How is nobody else seeing this?

Well, ... they are. This has been being discussed for a while. It came up a bit ago, many people were seeing it in games with a weird error about low VRAM which apparently wound up being linked to Intel's 13th and 14th gen processors. Exactly why is anyone's guess.

https://www.tomshardware.com/pc-components/cpus/nvidia-blame...

Level1Techs is relatively well-trusted, and they seem to have some contacts that can also corroborate these issues.

https://www.youtube.com/watch?v=QzHcrbT5D_Y

One idea: Get 500 watt power supply, it powers new chip fine with it's unlimited configuration.

500 watt power supply becomes less efficient over time, now it's not quite 500 watts, parts of the chip brown out. Undefined behavior.

Hmmm, I think this is unlikely. For one thing, most people's power supplies are quite wildly overspecced in gaming rigs since its genuinely pretty hard to find good small ATX PSUs these days. For another, even crummy low-quality PSUs tend to mostly work pretty well for a reasonably long time in most cases, so it'd be weird to see this so much in high-end gaming rigs with brand new CPUs.

I also don't know exactly what failure mode you are suggesting with this (voltage rail out of spec?) but I think unless it starts feeding too much voltage into the CPU it is probably unlikely to be able to cause damage. (Instability, sure.)

It also doesn't explain why Intel's 12th generation CPUs are still running strong, along with most of AMD's lineup, by comparison. Don't get me wrong, every SKU of CPU has its issues, but all of the major issues with other CPU lines we have explanations for; the issues with Intel Raptor Lake are thusfar unexplained. Intel has blamed motherboard manufacturers quite extensively, but so far the evidence of this is pretty uncompelling, because the failures can apparently also be seen in abnormally high numbers on motherboards that never had unreasonable configurations.

Having built a few power management circuits myself: this is not how PSUs degrade, at least not in the first 5 years, and generally not "ever" in most cases.

The PSU (notably in this case the M/B VRM, not the ATX PSU) can't really trigger brownouts. 1.05V is 1.05V, and if asked to do so, the PSU will deliver 1.05V. It will just get hotter while doing so over time (and specifically: capacitors degrading.) The voltage references are quite stable over time, and so are the window comparators that ensure "1.05V" is between, say, 1.035V and 1.065V. (Voltage references have essentially continuous engineering history back to the 50s≈60s, it's a "solved" problem.)

Varies by brand, some use better parts. I've definitely seen file servers that won't boot after 3 years, then I pull a few drives to lower the power needed and it boots. After another year or two I have to pull more drives.

Even with generously specced power supplies, server class hardware, redundant power supplies, and enough smarts to stage the drive spin ups to avoid a large peak inrush current when 16 drives try to spin up at once.

But GP's explanation still holds. The PSU will give the reference voltage, while being less and less efficient. Unable to deliver proper voltage (because it becomes less efficient with time), will not deliver any, until you remove the drives and it can deliver it? Did I misunderstand something ?

Now I'm thinking of these HPE over-over-over-specced PSUs I run and y'all got me worried.

Sort of. It's not just efficiency that's lost, but peak power. But if peak power was causing the CPU problems it wouldn't get better with a replacement CPU.

Sure power supplies provide accurate power ... till they can't. Unlike peak power that starts to degrade from new, the voltages will hold as long as the PS can manage it.

A bit sardonic, but if peak power is the problem, your PSU better be shutting off, since the alternative is that it'll catch fire at some point. (And it normally will shut off, since it has thermal protections. Those are boolean, not somehow lowering output power.)

And again, the ATX PSU doesn't really matter here, the M/B VRM does. Even if the ATX PSU has a failure mode where it delivers 10V instead of 12V — you can still make 1.05V out of 10V. Decoupling is spread all the way too, so short peaks don't explain a lot either.

Lastly, it's of course possible the VRM is misdesigned. But this issue is seen across mainboards from at least 3 vendors… what is the likelihood of all of them getting it wrong in a very similar way? The primary way for that to happen is if they are all implementing Intel's reference design, in which case it's Intel's fault again anyway and we've looped back to the origin of the discussion…

As for everyone getting the same thing wrong being unlikely, seeing how much 'inspiration' there is in the space, especially anything helping the race to the bottom on price... it's not unseen, as in a whole generation of motherboards caps blowing or leaking, bad TPM implementations... not saying your overall point is wrong but this part I don't know

The problem description says it happens gradually over time, but affects roughly 100% of the machines client and server alike. I have a hard time believing that everybody has underspecced and failing power supplies, especially on their servers.

that's just alderon_matt's assertion. wendell and GN have triaged this to approximately 10-25% of units, seems pretty consistent acros a whole bunch of different environments.

https://www.youtube.com/watch?v=gTeubeCIwRw

similarly, the oodle maintainers only found it to affect a small % of machines (but reliably, of course).

https://news.ycombinator.com/item?id=39486930

The alderon_matt thing is actually carefully worded to say affected machines invariably degrade... but they don't say how many machines are affected. It's deceptively worded, intended to be read the way you read it, but actually not saying that 100% of machines are affected either.

at the end of the day game developers are just people too, and they're equally frustrated with the process/not willing to cut intel slack. but like, this is shades of the Ryzenfall thing (which, remember, contrary to early takes did have several real severity 9.0 issues behind it, including a PSP jailbreak and a UEFI module signing bypass, as well as memory encryption bypass and some other goodies - AMD doesn't issue AGESA patches for "root password lets you do root things"). People have moved past the facts and into the "trying to generate noise to get attention" on levels that are not supported by the facts.

https://www.youtube.com/watch?v=QuqefIZrRWc

the Alderon Games assertion, specifically, is very aggressively/deceptively worded and is not supported by the broader facts, nor does he have any real idea what can or can't be patched. as wendell notes - all kinds of "unfixable" hardware bugs are indeed fixed in microcode, routinely. that's why it's there, because of the pentium FOOF bug.

this isn't to say there isn't a degradation problem, there clearly is something (or more than one something) going on. but the alderon games press release is very over-the-top compared to all the rest of the evidence. GN is pointing at 10-25% and this is very consistent across a number of different operators, boards, environments, etc. That's the number Dell is getting in their validation (depending on the department/how stringent they validate), that's the number Wendell is seeing in operation in datacenter error reports. And they're thinking possibly some kind of fab defect affecting those units, basically (oxidation of the vias).

https://youtu.be/5KHCLBqRrnY?t=2280

Sure, that might explain it, except when people get replacements things are improved, but then start getting worse. Some even are on their 3rd CPU.

What's making news is that Intel is starting (in at least one case) to deny RMA replacements after the first, despite shipping defective CPUs as replacements.

> Are these guys executing code paths that are normally not used?

Yes: games are notorious for having poor multithreading and hitting one core with very high load. And unlike servers where you're rarely the only workload, the rest of the CPU will be rather idle on a desktop/gaming system. This pushes the CPU to a much more "imbalanced" mode (putting all the power and heat into a very small area) than is common elsewhere.

I have these issues on a dedicated Postgres server running Linux and literally nothing else. It’s not just games, it’s just a lot of gamers buy these chips.

Silicon aging is a thing, especially at high voltage.

Reply on news.ycombinator.com