It seems to be a power delivery issue, and fortunately fixable if you disable all spread-spectrum and VRM power-saving options, but the Zen series seems a tricky beast.
I've had machine crashes triggered by using the "wrong" CPU scheduler under Linux. It's amazing, in a horrible way.
To my knowledge, AMD hasn't implemented any proactive measures that are as severe as Intel's AVX512 strategy, where the instructions get split and handled by the narrower vector units for a surprisingly long time while powering up the full-width vector units (and dropping CPU clocks). This AMD instability is only using AVX2: 256-bit SIMD rather than 512-bit. But spread across so many cores, that's still a lot of FPUs to be lighting up.
(Ok, it actually ramps up frequencies slower rather than lock out cores, but the effect is the same.)
https://travisdowns.github.io/blog/2020/01/17/avxfreq1.html
(specifically: https://travisdowns.github.io/assets/avxfreq1/fig-volts256-1... )
> So one theory is that this type of transition represents the period between when the CPU has requested a higher voltage (because wider 256-bit instructions imply a larger worst-case current delta event, hence a worst-case voltage drop) and when the higher voltage is delivered. While the core waits for the change to take effect, throttling is in effect in order to reduce the worst-case drop: without throttling33 there is no guarantee that a burst of wide SIMD instructions won’t drop the voltage below the minimum voltage required for safe operation at this frequency.
https://www.youtube.com/watch?v=HMUWzDSAS9c
A very interesting watch if you're interested in electronics in general and in power delivery in particular (the whole YouTube channel is awesome to be honest).
But I wonder if this is a Gigabyte issue who have a history of playing around with the power delivery and using "fake phases" (to the point where they now have to advertise their boards as having 16 "real phases"). As far as I can tell many (most?) reviewers benchmarked the board with 24 core CPUs and most likely skipped on the power intensive tests.
Do they just not believe in compiler authors to be able to emit these kinds of hints? Is there too much legacy code that would come without the hints for it to be worthwhile? Would it just not be worth it given how often OS context switches could drop the CPU directly from regular code in one process to AVX2 code in another process?
I have my doubts about this being a power issue, because localized power analysis is pretty advanced in IC design and they'd totally have tested for this.
Assuming it is a power issue though, nearly all power is used in dynamic switching, so simply inserting stall cycles will resolve it.
Power rails on silicon chips have quite a lot of capacitance, at least enough for many clock cycles, so you don't even need to make every cycle in every bit of hardware a possible stall cycle - it would be enough to simply gate instruction issue I expect and let functional units drain.
Another approach I haven’t seen from either Intel or AMD yet, is to copy ARM’s big.LITTLE architecture: to set up separate “AVX2 cores” that can execute AVX instructions and most basic ALU ops but not e.g. branches, put those on the other side of the die with in-wafer thermal insulation between them and the regular cores; and then throw workloads between regular and AVX2 cores in a way where the AVX2 cores heating up doesn’t mean that the non-AVX2 cores are heating up, and the CPU can go back to full Turbo Boost as soon as the workload is thrown back to the regular cores, because the respective regular core is actually quite cool.
(The performance effects of this are possible to loosely estimate, I think, by writing code that synchronously sets up a GPGPU pass on some data, executes it, retrieves the result, and then returns to executing CPU instructions for a while, in a loop; and then executing this code on an Intel CPU using its on-die IGPU as the GPU. The CPU and IGPU form somewhat-separate thermal domains already, though there’s no explicit insulation.)
Sorry to be pedantic, but... physics requires a degree of pedantry.
1. This is probably an ENERGY issue, not a power issue. Energy is the integral of power (or power is the derivative of energy: power is the change of energy over time).
2. Electronics can only really measure voltage easily. Everything else is converted to voltage to be measured. For example, to measure current, you create a resistor and then measure the voltage across the resistor. To measure power, you need to measure voltage, and multiply it with current. To measure energy, you take the integral of your power measurement. (No joke: that's how it typically works).
3. With a capacitor, you can measure your energy level. Capacitors are relatively inaccurate however (+/- 10% tolerances at the electronics level, and probably worse tolerances at the nanometer chipscale level). The voltage from a capacitor does roughly correlate to its energy level.
-----------
Issue #1: Capacitors suck at energy storage, but they're all that engineers have at these speed and sizes.
#2: Voltage is the only thing that can be measured reliably, but it takes a long time (more than a few nanoseconds, aka 10s or 100s of clock cycles) to measure voltage. Measuring other variables, such as "Energy" (voltage level of a capacitor), or "Power" (change of energy over time, or voltage * current), takes even longer to measure.
#3: Computers are fast. 4GHz computer means that you have 0.25 nanoseconds to make a decision per clock tick. Waiting on the results of any calculation on #2 will naturally take dozens or hundreds of clock ticks to accomplish.
#4: That's best case scenario, assuming that capacitors are large enough to actually hold the energy you need to perform your calculations. But in an AVX2 situation, where your CPU suddenly will use up more Power (energy/time) than expected, you need to communicate to your power-circuitry to increase the voltage or current delivery to the chip.
#5: Sending commands to the power-circuits / VRMs, or worse case to the Power Supply, can take many milliseconds (millions of nanoseconds) before the Power Supply responds. A 200W chip at 1.25V will draw 160 Amps of current (0.160 coulomb of electrons per millisecond). A 1 Farad capacitor will lose 0.16 volts per 0.16 coulombs of electrons drawn, and hint-hint, you can't physically fit a 1-Farad capacitor in a chip (https://qph.fs.quoracdn.net/main-qimg-dddab4aa574936f8a61437...).
#6: This means that the only reasonable approach with today's technology, is to wait for the VRMs or Power Supply to respond. Note: The Power Supply will naturally notice the voltage drop and automatically send more electrons down to the motherboard. But the time-scale the PSU operates is on the order of milliseconds. A millisecond is 1-million nanoseconds or ~4-million clock cycles AFTER the AVX2 units started getting used. If you run out of energy before the power gets to your chip... guess what? You get a brownout. Random cells in your chip were now turned off and the state of the CPU is a jumbled mess now.
--------
Where, how, and why to place capacitors around a chip is one of the most difficult PCB-issues an electronic engineer deals with. There's a LOT of "rules of thumb", making PCB-design / capacitors a mystic subject matter for many EEs.
Given the difficulty of the subject of power-delivery, energy calculations, and high-speed signals involved (a chip operates at the GHz band, drawing ~100 Amps on the clock-tick, and then 0 Amps between clock ticks!), I'm almost certain that there's a mistake in the capacitors somewhere.
Its a mystic subject that's very difficult to simulate, especially across varying workloads (AVX vs normal 64-bit code, etc. etc.). That's my bet for what's going wrong here.
Honestly, power-delivery is so awesome and complicated. I'm almost surprised that anything works at all these days.
But, if such "hint" ops were seen as the only viable way to get correct, high-performance behavior in these cases, and there was a commercial case to be made for it, I could see processor vendors adding them to the IA.
> I observed an interesting phenomenon when executing 256-bit vector instructions on the Skylake. There is a warm-up period of approximately 14 µs before it can execute 256-bit vector instructions at full speed. Apparently, the upper 128-bit half of the execution units and data buses is turned off in order to save power when it is not used. As soon as the processor sees a 256-bit instruction it starts to power up the upper half. It can still execute 256-bit instructions during the warm-up period, but it does so by using the lower 128-bit units twice for every 256-bit vector. The result is that the throughput for 256-bit vectors is 4-5 times slower during this warm-up period. If you know in advance that you will need to use 256-bit instructions soon, then you can start the warm-up process by placing a dummy 256-bit instruction at a strategic place in the code. My measurements showed that the upper half of the units is shut down again after 675 µs of inactivity.
https://www.agner.org/optimize/blog/read.php?i=628
Agner also notes that previous generations would just flatly stall all instructions (including non-vector) once they saw 256-bit instructions, which is probably why there is no "warmup" instruction. If there is an explicit stall of all instructions then there are very few reasons to incur that until the last possible second.
Everything in computer science is such a rabbit hole, great field to be in.
It was the first 'real' scope I bought and I still use it a fair amount despite having upgraded to a Rohde and Schwarz MSO model. I've been amazed how much I use an oscilloscope after getting one, from measuring ripple on power supplies to diagnosing serial communication issues.
Any recommendations for learning resources that could help with understanding DC power supply analysis for non-EE types? While refurbishing laptops and working with microcontrollers I’ve run into some odd things where ruling out transient power supply issues would probably be helpful.
As for learning resources, I came across a decent article on the subject when I was starting out (1), and most of the oscilloscope manufacturers have whitepapers on SMPS diagnostics, the Tektronics one I read a while back (2) gave a good overview. A lot of the whitepapers have a manufacturer-specific focus, but they still have good information that can be applied to almost any oscilloscope.
If you want to get really into the power supply and do high-side measurements you'll need an isolated differential probe, which can cost as much as an inexpensive oscilloscope, but for DC output measurement you shouldn't need anything special. Current probes are a lot more affordable if you're interested in looking at loads or current fluctuations/harmonics, but that's more useful after you've figured out a bit more what specific properties you're trying to measure.
1: https://www.testandmeasurementtips.com/test-switching-power-...
2: https://download.tek.com/document/3GW_23612_7.pdf
Edit: I forgot to mention that the EEVblog forums are a good resource also, but they sometimes aren't as friendly as they could be towards people just starting out.
It's interesting how Rigol did the locking of those extra features. The unlock key for a feature set for your scope has to be signed by a Rigol private key using an elliptic curve signature system.
But they are only using a 56 bit private key. That was quickly brute forced, and key generators proliferated.
They used a good library for the cryptography stuff, and except for the short key seem to have used it well and knew what they were doing. This suggests that the choice of a weak key was deliberate.
Each family of scopes has its own private key. As few families came out with new private keys, Rigol continued to use 56 bits. When major firmware upgrades came out in existing families, where they could have easily changed to a longer private key, they kept the same 56 bit that was now widely circulated on the net.
It seems pretty clear that they are not interested in stopping people from free unlocking.
p.s. lovely/lively oscilloscope shots!
If these are on by default, are they required for FCC certification conformance? Presumably the device has not been EMI/EMC tested in the mode where spread-spectrum was disabled.
The frequencies are up in the microwave band, so I can't imagine there's much range.
If it was designed well ;)
But that's only effective for addressing radiated emissions, not conducted ones.