Hacker News Reader

Comment by bdd | Hacker News Reader

“Unable to perform AVX2 instructions correctly under heavy load” is also a common “WTF Intel!?”–inducing phenomenon. I’m certain SREs who work at companies with more than 1 million servers have a bunch of hair pulling stories.

Most (all?) Intel server CPUs in fact decrease clock speed when executing AVX2 (and some other) instructions to keep things a bit more sane. Vlad from Cloudflare wrote about this, more specific to AVX-512 back in 2017: https://blog.cloudflare.com/on-the-dangers-of-intels-frequen...

Then there is PROCHOT signal. Which is supposed to protect the CPU from getting too hot but keeps getting raised in lopsided AVX2 loads not because CPU is too hot but voltage regulation gets whacked.

You may wonder: what is an example of AVX2 heavy load. RSA multiplication is a good candidate. AES constructions or modes (CBC with SHA, GCM) are implemented in AVX2-BMI2 as well.

View on HN

I ran into this issue on one of my builds. Aida64 has a benchmark (floating point photo or something?) that uses AVX instructions. Pressing the "run benchmark" button would instantly black-screen crash my machine with 100% certainty.

I debugged this problem over a number of years... I replaced the RAM, I replaced the motherboard, I eventually replaced the CPU... still happened no matter what I did. Even if I underclocked the machine and kept the voltages the same, instant crash. It was maddening.

Exasperated, I eventually busted out an oscilloscope and looked at the waveform on the 12V supply to the CPU. When starting the AVX benchmark, there was a huge brownout. That basically explained everything; my power supply essentially turned off when the CPU started drawing a ton of power. I replaced the power supply and got lucky -- it handled it fine and I could run the benchmark. I even got some overclock out of it.

After this whole experience, I've never looked at computers the same way again. You can buy high-spec brand-name components, and it's all just a crapshoot. Maybe your computer won't crash in the middle of an important task. Maybe it will. There isn't much you can do but cross your fingers.

If there is one thing in a computer you should not cheap out on, it is the power supply. Buy from Seasonic and buy more power supply than you currently need.

My bad power supply was a Corsair 1300W model (noted because others in the thread are offering them glowing praise, whereas I think they're just a nice logo on whatever is cheap this week in Shenzhen.) The working one is indeed a Seasonic ;)

I overbuy power supplies because I think, but have not tested, that it will convince them to run the fan at lower speed. This strategy has resulted in quiet, for sure, but I'm not sure that it gets you reliability.

> I overbuy power supplies because I think, but have not tested, that it will convince them to run the fan at lower speed. This strategy has resulted in quiet, for sure, but I'm not sure that it gets you reliability.

This theory is sound. High-end power supplies tend to not turn on the fan until reaching 30-50% of their rated maximum output. Fanless power supplies are usually internally using the same components as regular high-end models with 80-100% higher output ratings, but with larger heatsinks. A high-quality oversized power supply with a fan will be silent most of the time, but keep itself cool when necessary.

( I have a semi-passive 750W power supply that has spun up its fan exactly once, when I accidentally shorted two of its power rails together and melted some wires while burning out the PCIe slot I was measuring the power draw through. The motherboard and wires had to be replaced, but the CPU, PSU and PCIe card were all fine.)

This. I've had great luck with Corsair (who used to source from Seasonic) and after some of my experiences with less quality sources (Thermaltake....eww) I'll never go back.

Spending days trying to chase down an intermittent reboot just to find that your 750W PSU is actually shitting the bed around ~350W is a terribly unsatisfying journey.

this is not universally true. i bought a new 1000W seasonic prime ultra titanium for use in a hashcat rig not too long ago, and the PSU would immediately switch off when all the cards drew power to 100% (all the cards drew no more than 800W or so at load.) there is a newegg reviewer that experienced similar issues in a system with dual RTX cards. ended up buying an EVGA supply instead and that has been solid, even at 24x7 operation.

I always spring for an 80+ Gold or better efficiency rating. My theory is that higher efficiencies require tighter manufacturing tolerances, which means more care has to go into manufacturing and therefore you get a better quality as a result. That theory has yet to fail me.

Is Corsair typically a reliable brand? I usually use them for desktop builds but simply out of habit.

Corsair used to (and still might, but I haven't needed to replace my power supply in many many years) use Seasonic as their OEM. So for a long time, Corsair was just rebranded Seasonic.

These days I don't know if that's entirely true, so I would at least look at reviews to make sure.

I'm not sure if any of Corsair's current high-end PSUs are Seasonic; they've definitely done some that aren't. But whether they're using Seasonic or not, Corsair's fairly involved in the design process and set their own specs rather than just taking an ODM's existing design and adding their own logo. They also have a serious test lab where they do their own design validation, and I believe they also do failure analysis on returned products (since that lab's only a few rooms away from where RMAs land).

I'm sure these days they're far more involved in the creation of their power supplies than they were a decade ago when I last had a need to really research any of this.

Anecdotally, Corsair has always been a top-notch company to me. Even when their products go bad, their support is awesome. I had a set of memory sticks appear to go bad, but I couldn't tell if it was my northbridge or the RAM itself. I was a cash-strapped kid, so I didn't have another machine to test with. Corsair sent out a set of known-good RAM sticks for me to test with, at completely no charge to me. When the test revealed that it was indeed the RAM that was going bad, they offered to let me keep the test sticks they sent me until my replacement came in the mail.

https://www.orionpsudb.com/corsair

The OEM for most of their PSUs is CWT, but they are "Corsair custom" designs instead of off-the-shelf. Seasonic is only used for relatively few of Corsair's models.

Ah, my information is a bit out of date because thanks to the quality of Corsair power supplies, I haven't actually needed to buy a new one in a long time.

Corsair is good, I have been using them for high end desktops and low end servers for quite some time.

Only one out of ~ 10 I bought ever failed on me, and that was after 2 years of no usage.

Corsair's higher end stuff is good, yes.

You say not to cheap out, and then recommend Seasonic. I'd say avoid a brand that was notoriously the cheapest option in recent history.

Some Seasonic are probably among the best consumer power supplies you can find (e.g. https://www.anandtech.com/show/11252/the-seasonic-prime-tita...)

At the very least, Anandtech seems to rate Seasonic very highly: https://www.anandtech.com/show/11252/the-seasonic-prime-tita...

It's not just Anandtech. All reviewers have nothing but praise for Seasonic. They have the best reputation in the industry at the moment and all their PSUs have a 10 year warranty to back it up.

Jonnyguru's last Seasonic review: https://www.jonnyguru.com/blog/2018/07/03/seasonic-prime-ult...

Literally nothing in the bad & mediocre summary sections, and with nothing but recommendations for the entire series:

> Our look at the PRIME Ultra Titanium series has now at last come to a conclusion. We can now definitively say there’s not a bad performer in the bunch, and while all do have some very minor drawbacks I can’t see any reason not to recommend any of them.

They actually have 12 year warranty at least for the PRIME TX ones.

You'd be surprised to hear that part from their own brand (which is excellent) many other top tier power supplies from the best brands are actually made by Seasonic.

A lot of brands, Seasonic among them, vary from model to model. The last time I felt like I could just blindly stick with a brand was PC Power & Cooling, and they've since been absorbed by OCZ, so... ?

I had weird issues a few times, and it was always the power supply. Before I would never think of it as the responsible for issue, but now I always do. Each time a friend has a weird unexplainable problem with a gaming PC, I suggest to try another power supply and it always fixed the issue.

For me it's the same, but with one additional step.

Test the RAM in a known stable machine. If it passes an overnight memtest, then replace the flaky machine's PSU.

RAM has always been my "go to" "I bet this is it" problem, but I've debugged my own machines and other people's machines and I haven't seen bad RAM in ages.

I avoid XMP, however, and typically manually specify the voltage that is listed in the RAM's spec sheet and run at the speed that the CPU manufacturer recommends. (So the RAM might be sold as 3200 but Intel says 2400 is the max, so I run it at 2400.)

I get your desire to be conservative here, but this is not good practice with modern CPUs. For example AMD desktop CPUs are designed with DDR4 speeds well north of 3000MHz in mind[1]. Problem is that JDEC profiles simply don't go that hight for some reason, so you you have to run with XMP profiles if you want the chip to perform as the manufacturer intended.

[1] https://www.anandtech.com/show/14525/amd-zen-2-microarchitec...

I look at Intel ARK and use what they say the CPU is rated for. I am a couple generations behind, so not at the bleeding edge. (Haven't used an AMD CPU, but I would find their equivalent if I did.)

Keep this in mind for when you do: The first boot will be in a "failsafe" 2133MT configuration, but the fabric clock is intended to ramp all the way up to 1800MHz.

You need 3600MT to match that while keeping 1:1 sync, which means XMP unfortunately.

You can do a little bit to help by ensuring you shop for the right kind of high-spec: workstation rather than gaming, so that the extra money goes toward useful engineering and QA rather than RGB LEDs. But even then, it often ends up that the best you can hope for is a long warranty with a quick and easy replacement process.

I've had enough failed server and workstation components to believe this is snake oil. It seems like a lot of "Enterprise" hardware (particularly hard drives) is mostly the same chipset/components with a longer warranty and a more boring PCB color. Or more annoyingly, differentiated by feature lockout in some firmware.

Gaming hardware has larger economies of scale which leads to more people running the hardware and submitting RMAs. Gamers are also more likely to overvolt their hardware, leading to most consumer components being overspec'ed for heat and power.

As such, the second revision of a gaming board imo is more refined than a workstation/server board that almost never gets a new revision.

On a more practical level, consumer CPUs usually have higher clockrates and newer architectures and gaming NVidia GPUs are so much cheaper than the enterprise product lines that you can buy multiple. My workflow is faster with the cheap stuff, sometimes twice as fast.

To some extent, this depends on what kind of component you're talking about. I agree with what you're saying about consumer vs workstation GPUs because the price disparity is so large, but consumer GPUs don't usually get the full RGB treatment anyways.

I think you're also overestimating how big the gaming desktop components market is, and underestimating how fragmented it is. For motherboards, the economies of scale apply to the OEM custom motherboards used by Dell, HP, etc., which have far more volume than any one consumer retail motherboard model. You're probably also vastly overestimating how many gaming-oriented motherboards make it past revision 1.0. I just checked Gigabyte's top gaming product line, and out of 75 models listed, all but 8 were revision 1.0. Browsing through a similar number of their most recent mainstream consumer motherboards didn't turn up any that went beyond rev 1.0.

Some gamer-oriented motherboards are over-specced for power delivery and cooling. It's definitely common to find VRM heatsinks that are oversized gaudy sculptures, but that doesn't help product reliability. There are also tons of examples of gamer-oriented boards that advertise a large number of VRM phases, but have obvious shortcomings in their power delivery design once the heatsinks are removed and an EE takes a close look at things. And then there's the motherboard at issue here. When gamer boards are overbuilt, it's not thorough, it's usually just the most visible aspects that get beefed up (and the price).

RAM is a really obvious area where paying a bit extra for workstation-class ECC DIMMs gets you a real reliability advantage, but paying extra for high-speed consumer modules with RGB LEDs and big heatsinks does not.

RGB LEDs are still thankfully pretty uncommon on SSDs, but the handful of examples show that the impact varies from compromised performance to catastrophic instability. Most gamer-oriented SSDs with no LEDs but with flashy heatsinks also fail to outperform or outlive the best pedestrian-looking models.

Enterprise and NAS hard drives are definitely diverging from mainstream consumer hard drives in more ways than simple firmware feature gating. Some of the technologies they're using to drive density beyond consumer HDD needs have a questionable impact on reliability, but filling them with helium does help reliability, and you don't find that on a WD Black or Seagate Barracuda.

Oh, WD Purple surveillance HDD brochures are just marketing bullshit™:

"Using AllFrame™ technology, WD Purple™ drives improve video capturing and help to reduce errors, pixelation, and video interruptions that can occur in a video recorder system."

"WD Purple 8TB, 10TB, 12TB & 14TB capacities feature AllFrame AI technology that enables not only recording up to 64 cameras, but also supports an additional 32 streams for Deep Learning analytics within the system."

"WD Purple drives with AllFrame AI technology feature a workload rating up to 360TB/year to support the Deep Learning analytics that is featured in AI capable NVRs."

https://documents.westerndigital.com/content/dam/doc-library...

Good call. After learning the hard way in my youth, now when I build a machine I don't try to save money on the power supply. Instead I do a bunch of research, stick to the premium models, and I haven't regretted it.

I would say this is why burn-in has been ingrained into me as extremely important, especially target workload specific burn-in. I see way too many cots things cobbled together and then cursed for failing in some scenario that should have already been tested for (and specifically built for).

Prime95 has been a bad test for years now. Literally since the introduction of AVX2, it has been known that overclocking and running Prime95 will cause rapid electromigration and processor degradation on Haswell-E if you attempt to overclock. It's fine for like a few minutes but you should not attempt to do 24 hour Prime95 runs like people used to do, that was in the days before AVX was a thing.

https://rog.asus.com/articles/usa/rog-overclocking-guide-cor...

It also does not test all parts of the processor equally, it really just is slamming the cache and the AVX units. You can be "Prime95 stable" and still crash in other things. It's just not a good test anymore in multiple respects.

Same thing for Furmark on GPUs. And nowadays GPUs are actually smart enough to realize they're running a power virus and the power management pulls back the power. So again, it doesn't demonstrate anything. Zen2 is supposed to work the same way, if it realizes you're redlining the chip then it should be clocking down somewhat.

No sequence of instructions should cause the machine to reboot.

That's an excusable outcome if you're overclocking or otherwise tweaking critical operating parameters beyond their safe default ranges.

Except the 'reboot' command? I'm always curious how it works.

Normally you'd ask the chipset so that it can reset all the peripherals at the same time.

On Intel you can also intentionally "triple fault" the CPU and that'll cause it to reset. The way that works is you tell the CPU that it doesn't have interrupt handlers anymore, then cause a software interrupt. Then the CPU will first try to call your interrupt handler (but that'll fail because you told it there are no handlers), then errors during an interrupt cause a "double fault" interrupt that's supposed to handle those errors (but once again, there's no handlers), so now you're in a "triple fault" condition and the only way to recover is to reset, so that's what the CPU does.

"Power virus" is a propaganda term invented by hardware manufacturers as an excuse to avoid fixing their defective hardware.

Designing chips that can be safely run at redline 24/7 will inevitably involve a significant reduction in voltage (and thus clocks) for the 99.99999% of users who aren't running them at redline 24/7.

Would you buy a GPU that you could safely run Furmark on 24/7 if it was limited to, half or 3/4 of the clocks? That's basically what you're proposing. I seriously doubt that you would actually lay money down for that shit. The market as a whole certainly wouldn't, they would buy the card that's faster under actual real-world workloads and/or didn't require manual overclocking to achieve its potential in real-world workloads.

The good news is that you can have it both ways. Modern power management is smart enough to figure out when you're running a power virus and it will downclock to those safe levels, while us normies can get competitive clocks when running our real-world code. It works pretty well most of the time.

> Would you buy a GPU that you could safely run Furmark on 24/7 if it was limited to, half or 3/4 of the clocks?

Given that modern GPUs have complex frequency scaling algorithms that take temperature and power usage into account when setting clock speed, there's a good chance that a modern GPU will do exactly that.

> Modern power management is smart enough to figure out when you're running a power virus and it will downclock to those safe levels, while us normies can get competitive clocks when running our real-world code.

Which is perfectly fine, as long as the clock speed behavior is disclosed (which it is).

What isn't fine is a product running at such a high speed from the factory that running certain workloads damages it.

If you look back at the OP, he specified "while overclocking".

I believe that's where the issue lays. Overclocking usually also means increasing the voltage.

>electromigration and processor degradation on Haswell-E

So what you are really saying is Intel made another defective product.

No, I'm just using that as an officially documented example of Prime95 being problematic.

AMD is subject to electromigration as well. I've seen a couple guys on Reddit already degrade their Zen2 chips from pushing 1.4V through them just during normal usage, not even Prime95. Literally weeks of usage.

Anti intel circlejerk ahoy.

That sounds like it’s a great test actually.

What part of damaging your processor due to current draw, while not actually demonstrating any particular level of processor stability, sounds like a great test?

Nice contrarianism.

Yes, you can run it, and it should run successfully. Yes, sustained testing will rapidly damage your processor and cause notable performance degradation (reduced maximum overclocks/increase in required voltages) within a matter of days, and yes, you can still crash in Aida64 or other stress tests because it's not testing the whole processor very well.

It's far outside the normal current draw, even compared to other AVX2 loads like video encoding. It's an AVX load that fits entirely inside instruction cache and will hammer the AVX2 units nonstop at the maximum possible rate with absolutely no other instructions going to any of the other units. That's not how any other workload behaves, even video encoding. CPUs are not designed around that, they're designed for some reasonably normal mix of instructions.

Yes, it's a good idea to stress your processor a bit more than normal to prove it's stable, but if you make your 'proof charge' too large you will still damage the firearm.

I don't understand, when I buy a processor I expect it to work 100% load for all of the instruction set. Anything less is a defective product, and should be returned.

Furthermore, it should be able to do this 24/7 for years at a time.

This test, assuming it actually does kill the processor or stops executing, clearly indicates it should be returned.

It's fine at normal clocks. Not good for the chip, but it won't kill it within the normal operating lifetime (say, 3-5 years). That's why chips ship with a little more voltage than they need, so that they can suffer some electromigration and still function.

Overclocking? Yeah, it'll kill it quick, and that's your problem, because you're operating it outside normal specifications (more voltage than designed). If that's your standard, that you should be able to overclock and then redline the chip 24/7 for an unlimited period of time and it should never kill the chip, then you're going to be disappointed.

The fact that Threadripper is having problems is definitely a fault but it's something that you really shouldn't be doing anyway (and it sounds like the problem is the motherboard not the processor).

More generally: this is like arguing that you should be able to buy a camry and run redlined 24/7 and get the inevitable engine failures covered by warranty. If the manufacturer finds out you're doing that, they're not going to cover it, because it's not normal usage. Somehow people think it's a reasonable expectation for processors.

I agree Prime95 is overkill, but a processor should be able to handle it.

This is like the cooling argument all over again - "your cooling system is good enough for normal workloads". No, it must be good enough for permanent full load! Like a CPU under Prime95. If your heatsink lets it overheat, it's defective. You saved a few pennies on the copper and now it's a defective product.

Same for Prime95, the CPU must be able to run it for days at end and last ~5 years. Doesn't seem like too much to ask for, imo. Intel says their chips will run at 98 degrees for years, so why would a full load like Prime95 kill them in days?

One other thing, overclocking these days is actually undervolting combined with raising clocks. The old school "just raise the voltage" is a bad way to do it on any Intel chips post-Sandy Bridge...

If you overclock CPU, you lose warranty. Even if it's K-series which is made for overclocking. And stock settings probably are safe.

Yeah I did mean at stock settings.

If you overclock, you're responsible. Losing warranty sounds acceptable to me. Better than manufacturers doing everything to make overclocking impossible.

To me, it's a great way to get more performance out of an old chip, the warranty would be expired at that point.

It is hard to prove a CPU was overclocked though, so I can see some people taking advantage of that.

Did you notice that in the original article he is running the stock clock speed? If a CPU can't handle any instructions it supports at stock clock and voltage for any period of time it is defective, full stop.

Change the word "processor" for "car" and the absurdity of your argument should be fairly obvious. Most ordinary cars will provide reliable transport for many years, but will literally catch fire after a few hot laps on a racetrack. It makes no sense to design a passenger car with brakes, suspension, cooling and tyres that are massively over-engineered for normal use just in case the owner wants to go endurance racing.

3970x is a race car in your analogy, and thus should not catch fire

Damaging the processor? A CPU being unstable while overclocked doesn't mean it is damaging the CPU.

This is a deep dive on frequency scaling and IPC throttling related to AVX512 instructions. The consequences are quite large, surprisingly complicated, and persist for an eternity, which is why you really have to coax the compiler if you want it to issue these instructions. https://travisdowns.github.io/blog/2020/01/17/avxfreq1.html

Discussion here: https://news.ycombinator.com/item?id=22077974

The newest AMD CPUs decrease clock speed based on parameters like heat, which AVX2 under heavy load will cause. So, AMD also decreases clock speed when executing AVX2, indirectly. Though in a very different fashion, continuously, rather than a distinct mode.

Intel's TurboBoost is such a marketing mess. It used to be more sane when they stated base clocks - the guaranteed frequency you pay for. Anything above was just extra performance when your CPU is not running hot/at power limits.

Nowadays it's just "up to XX GHz", and people expect it to run at those clocks.

AVX really pushes the CPU, an Intel Core/Xeon will hit the power draw limit pretty fast. They don't decrease clock speed as much as fall back to base clocks. The ones you paid for. Anything above that is just a bonus.

That said, you should never trigger PROCHOT even under full stress load with AVX! If you do, you need better cooling.

It's a last resort throttling feature for when your processor is hot enough to boil water :/

I do prefer Intel's solution of slowing down over AMD's solution of crashing. I guess the good news is if this does turn out to be a power delivery related bug, it's fixable with firmware.

Maybe it wasn't very clear in my comment but "correctness issues" more-or-less literally mean the instruction's result is incorrect. Like you ask 2*2 and it gives you 0xfff1309f. So depending on how you rely on that result, you may crash. Extending on my examples, bad RSA multiplication means failed authentication. Bad AES construction means encrypting things to garbage, impossible to decrypted by the recipient. So, let's say if you rely on a result from such instruction to do memory references, you're definitely going to crash.

Slowing down and producing the correct answers is clearly superior to crashing and/or incorrect answers.

I still don’t understand what it is about avx2 that results in these kinds of issues - is it really just a matter of increased number of execution units running at once causing weird power and heat issues?

> is it really just a matter of increased number of execution units running at once causing weird power and heat issues?

That's been the root cause of all the previous AVX weirdness I'm aware of. It's probably the case for these Threadrippers, too. Modern CPUs have very high power density and run at pretty low voltage, which results in insane current delivery requirements.

Not increased number of execution units, but increased number of transistors, yeah.

A normal multiply does 1 number at a time (32 bit for simplicity). AVX2 can do up to 8 multiplications at a time. That's a huge amount more transistors firing all at once and that causes the voltage to start to droop.

AVX-512 takes that even further and now it's 16 multiplications per unit, oh and Intel moved from 1x256-bit unit per core on Haswell/Broadwell to 2x512-bit units on Skylake-X, so it's potentially 32 multiplications at a time - 4x as much as AVX2.

Basically to prep for that much power being drawn all at once, the chip has to switch to a higher-voltage mode to account for the voltage droop caused by all those transistors switching at once in one place. It takes time for the regulator (on-chip Fully Integrated Voltage Regulator or motherboard VRM) to wind up the voltage far enough.

https://travisdowns.github.io/blog/2020/01/17/avxfreq1.html

https://travisdowns.github.io/assets/avxfreq1/fig-volts256-1...

At this level behavior is intensively analog and thermals/voltage both significantly affect transistor current draw and switching time, which feeds back into thermals and power consumption/voltage droop.

This gets even more problematic on 7nm/10nm type nodes and especially in GPUs where you are doing a huge amount of vector arithmetic all the time. Essentially it is no longer possible to design processors that are 100% stable under all potential execution conditions, or even under normal operating conditions, so you have to have power watchdog circuitry that realizes when it's getting close to brownout/missing its timing conditions and slows itself down to stay stable. That's why AMD introduced clock stretching in a big way with Zen2 (despite the fact that it's nominally been around since Steamroller). NVIDIA piloted this with Pascal, AMD piloted it with Vega and brought it to CPU with Zen2. You simply cannot design the processor to be 100% stable at competitive clocks anymore. You have to have power management that's smart enough to withstand small transient power conditioning faults.

https://semiengineering.com/managing-voltage-drop-at-107nm/

https://semiengineering.com/power-delivery-affecting-perform...

https://www.realworldtech.com/steamroller-clocking/

I’m curious if this behavior defined by something in hardware, microcode, boot-time BIOS flags, or higher level kernel/hypervisor/application code.

on unlocked intel cpus you can change the avx multiplier in the bios.

Oh neat (I haven’t messed with over clocking in years) - is it just avx that you can tailor? (Beyond the old school bus multipliers)

You know I think that cloudflare blog post is a great example of the engineering approach behind boringssl. It's optimized for actual workloads, where you decrypt or authenticate a short message and then move on to different activities, and engaging AVX512 doesn't actually pay off in reality. OpenSSL is optimized to produce the biggest number from `openssl speed` so of course in that light it makes perfect sense to enable AVX512. But if you're trying to use these libraries in realistic workloads you will begin to appreciate the boringssl approach.

Reply on news.ycombinator.com