AMD Threadripper 3970X under heavy AVX2 load: Defective by design?

On Intel you can also intentionally "triple fault" the CPU and that'll cause it to reset. The way that works is you tell the CPU that it doesn't have interrupt handlers anymore, then cause a software interrupt. Then the CPU will first try to call your interrupt handler (but that'll fail because you told it there are no handlers), then errors during an interrupt cause a "double fault" interrupt that's supposed to handle those errors (but once again, there's no handlers), so now you're in a "triple fault" condition and the only way to recover is to reset, so that's what the CPU does.

Furthermore, it should be able to do this 24/7 for years at a time.

This test, assuming it actually does kill the processor or stops executing, clearly indicates it should be returned.

Given that modern GPUs have complex frequency scaling algorithms that take temperature and power usage into account when setting clock speed, there's a good chance that a modern GPU will do exactly that.

> Modern power management is smart enough to figure out when you're running a power virus and it will downclock to those safe levels, while us normies can get competitive clocks when running our real-world code.

Which is perfectly fine, as long as the clock speed behavior is disclosed (which it is).

What isn't fine is a product running at such a high speed from the factory that running certain workloads damages it.

I think you're also overestimating how big the gaming desktop components market is, and underestimating how fragmented it is. For motherboards, the economies of scale apply to the OEM custom motherboards used by Dell, HP, etc., which have far more volume than any one consumer retail motherboard model. You're probably also vastly overestimating how many gaming-oriented motherboards make it past revision 1.0. I just checked Gigabyte's top gaming product line, and out of 75 models listed, all but 8 were revision 1.0. Browsing through a similar number of their most recent mainstream consumer motherboards didn't turn up any that went beyond rev 1.0.

Some gamer-oriented motherboards are over-specced for power delivery and cooling. It's definitely common to find VRM heatsinks that are oversized gaudy sculptures, but that doesn't help product reliability. There are also tons of examples of gamer-oriented boards that advertise a large number of VRM phases, but have obvious shortcomings in their power delivery design once the heatsinks are removed and an EE takes a close look at things. And then there's the motherboard at issue here. When gamer boards are overbuilt, it's not thorough, it's usually just the most visible aspects that get beefed up (and the price).

RAM is a really obvious area where paying a bit extra for workstation-class ECC DIMMs gets you a real reliability advantage, but paying extra for high-speed consumer modules with RGB LEDs and big heatsinks does not.

RGB LEDs are still thankfully pretty uncommon on SSDs, but the handful of examples show that the impact varies from compromised performance to catastrophic instability. Most gamer-oriented SSDs with no LEDs but with flashy heatsinks also fail to outperform or outlive the best pedestrian-looking models.

Enterprise and NAS hard drives are definitely diverging from mainstream consumer hard drives in more ways than simple firmware feature gating. Some of the technologies they're using to drive density beyond consumer HDD needs have a questionable impact on reliability, but filling them with helium does help reliability, and you don't find that on a WD Black or Seagate Barracuda.

Only one out of ~ 10 I bought ever failed on me, and that was after 2 years of no usage.

These days I don't know if that's entirely true, so I would at least look at reviews to make sure.

This theory is sound. High-end power supplies tend to not turn on the fan until reaching 30-50% of their rated maximum output. Fanless power supplies are usually internally using the same components as regular high-end models with 80-100% higher output ratings, but with larger heatsinks. A high-quality oversized power supply with a fan will be silent most of the time, but keep itself cool when necessary.

( I have a semi-passive 750W power supply that has spun up its fan exactly once, when I accidentally shorted two of its power rails together and melted some wires while burning out the PCIe slot I was measuring the power draw through. The motherboard and wires had to be replaced, but the CPU, PSU and PCIe card were all fine.)

I avoid XMP, however, and typically manually specify the voltage that is listed in the RAM's spec sheet and run at the speed that the CPU manufacturer recommends. (So the RAM might be sold as 3200 but Intel says 2400 is the max, so I run it at 2400.)

It was the first 'real' scope I bought and I still use it a fair amount despite having upgraded to a Rohde and Schwarz MSO model. I've been amazed how much I use an oscilloscope after getting one, from measuring ripple on power supplies to diagnosing serial communication issues.

I have my doubts about this being a power issue, because localized power analysis is pretty advanced in IC design and they'd totally have tested for this.

Assuming it is a power issue though, nearly all power is used in dynamic switching, so simply inserting stall cycles will resolve it.

Power rails on silicon chips have quite a lot of capacitance, at least enough for many clock cycles, so you don't even need to make every cycle in every bit of hardware a possible stall cycle - it would be enough to simply gate instruction issue I expect and let functional units drain.

Another approach I haven’t seen from either Intel or AMD yet, is to copy ARM’s big.LITTLE architecture: to set up separate “AVX2 cores” that can execute AVX instructions and most basic ALU ops but not e.g. branches, put those on the other side of the die with in-wafer thermal insulation between them and the regular cores; and then throw workloads between regular and AVX2 cores in a way where the AVX2 cores heating up doesn’t mean that the non-AVX2 cores are heating up, and the CPU can go back to full Turbo Boost as soon as the workload is thrown back to the regular cores, because the respective regular core is actually quite cool.

(The performance effects of this are possible to loosely estimate, I think, by writing code that synchronously sets up a GPGPU pass on some data, executes it, retrieves the result, and then returns to executing CPU instructions for a while, in a loop; and then executing this code on an Intel CPU using its on-die IGPU as the GPU. The CPU and IGPU form somewhat-separate thermal domains already, though there’s no explicit insulation.)

Sorry to be pedantic, but... physics requires a degree of pedantry.

1. This is probably an ENERGY issue, not a power issue. Energy is the integral of power (or power is the derivative of energy: power is the change of energy over time).

2. Electronics can only really measure voltage easily. Everything else is converted to voltage to be measured. For example, to measure current, you create a resistor and then measure the voltage across the resistor. To measure power, you need to measure voltage, and multiply it with current. To measure energy, you take the integral of your power measurement. (No joke: that's how it typically works).

3. With a capacitor, you can measure your energy level. Capacitors are relatively inaccurate however (+/- 10% tolerances at the electronics level, and probably worse tolerances at the nanometer chipscale level). The voltage from a capacitor does roughly correlate to its energy level.

-----------

Issue #1: Capacitors suck at energy storage, but they're all that engineers have at these speed and sizes.

#2: Voltage is the only thing that can be measured reliably, but it takes a long time (more than a few nanoseconds, aka 10s or 100s of clock cycles) to measure voltage. Measuring other variables, such as "Energy" (voltage level of a capacitor), or "Power" (change of energy over time, or voltage * current), takes even longer to measure.

#3: Computers are fast. 4GHz computer means that you have 0.25 nanoseconds to make a decision per clock tick. Waiting on the results of any calculation on #2 will naturally take dozens or hundreds of clock ticks to accomplish.

#4: That's best case scenario, assuming that capacitors are large enough to actually hold the energy you need to perform your calculations. But in an AVX2 situation, where your CPU suddenly will use up more Power (energy/time) than expected, you need to communicate to your power-circuitry to increase the voltage or current delivery to the chip.

#5: Sending commands to the power-circuits / VRMs, or worse case to the Power Supply, can take many milliseconds (millions of nanoseconds) before the Power Supply responds. A 200W chip at 1.25V will draw 160 Amps of current (0.160 coulomb of electrons per millisecond). A 1 Farad capacitor will lose 0.16 volts per 0.16 coulombs of electrons drawn, and hint-hint, you can't physically fit a 1-Farad capacitor in a chip (https://qph.fs.quoracdn.net/main-qimg-dddab4aa574936f8a61437...).

#6: This means that the only reasonable approach with today's technology, is to wait for the VRMs or Power Supply to respond. Note: The Power Supply will naturally notice the voltage drop and automatically send more electrons down to the motherboard. But the time-scale the PSU operates is on the order of milliseconds. A millisecond is 1-million nanoseconds or ~4-million clock cycles AFTER the AVX2 units started getting used. If you run out of energy before the power gets to your chip... guess what? You get a brownout. Random cells in your chip were now turned off and the state of the CPU is a jumbled mess now.

--------

Where, how, and why to place capacitors around a chip is one of the most difficult PCB-issues an electronic engineer deals with. There's a LOT of "rules of thumb", making PCB-design / capacitors a mystic subject matter for many EEs.

Given the difficulty of the subject of power-delivery, energy calculations, and high-speed signals involved (a chip operates at the GHz band, drawing ~100 Amps on the clock-tick, and then 0 Amps between clock ticks!), I'm almost certain that there's a mistake in the capacitors somewhere.

Its a mystic subject that's very difficult to simulate, especially across varying workloads (AVX vs normal 64-bit code, etc. etc.). That's my bet for what's going wrong here.

Honestly, power-delivery is so awesome and complicated. I'm almost surprised that anything works at all these days.

High end in consumer stuff, and especially anything gaming related, is basically scam.. look mah, 16000 dpi! And blingy leds! (But no engineering)

Sure they might drop an over-specced part somewhere in it but it's just marketing when the rest of the product is crap and still has no proper engineering behind it.

It's incredibly frustrating. I think there was a time when you could generally assume that expensive = high end = actually good, but now it's just a cheap thing with crazy markup and a premium part (but nowhere near premium enough to justify the markup) or two somewhere (where it probably doesn't matter much anyway) along with other gimmicks. Now it's just expensive = expensive, good or maybe not.

Part of it was likely that NVIDIA's performance was so good. Some anecdotal rumors from the AMD vlogosphere suggest that AMD thought that the RX 480 would be competitive with the GTX 1080 - I consider this kinda dubious because the RX 480 is basically a GTX 980 tier chip, so that would mean AMD thought that NVIDIA wouldn't make any progress at all from a node shrink, which seems dubious. Maybe they figured the RX 480 would be a lot faster or more efficient than it actually ended up being.

Polaris was the last generation with "old style" voltage control where you just set a target and go, it is possible that they figured the chips would hit close to 2 GHz like NVIDIA's but ended up with validation problems and didn't hit the expected clocks. This could possibly be the reason they nuked Big Polaris and ended up lengthening the pipeline so much in Vega to try and get the clocks up (along with adding the Pascal-style "smart" power conditioning management).

Anyway, regardless, the point is that it seems likely that AMD was forced to push the RX 480 much farther than they intended to, at the last moment. Like, after the PCBs had already gone out for manufacturing, and it was too late to switch the 6-pin to an 8-pin (which would have solved the whole problem, that would have allowed 225W rated total board power).

It was still a bit of a showoff move to run from only a 6-pin, there is no way that the card would have been significantly less than 150W total board power, but perhaps defensible as a compatibility move - although some marginal PSUs still might not have handled it.

Add.: OK, I finally found a couple of "Precision Boost Overdrive" menus deep down in the overclocking settings of this board's BIOS, and I've disabled everything PBO I can find while letting the "core performance boost" remain in auto mode. Without doing any extended testing this seems to have solved the problem, as I can now let Prime95 run through the AVX2 code path with 6 workers without any crashes. Thanks for the hint!