OP doesn't seem to show any major problems in his microbenchmarking, which is a start.
Daniel has reviewed this problem in series of posts: - https://lemire.me/blog/2018/08/15/the-dangers-of-avx-512-thr... - https://lemire.me/blog/2018/08/25/avx-512-throttling-heavy-i...
Below L1, there is an even lower frequency tier, L2, which behaves as you say: it doesn't drop to L2 unless there is a sustained use of heavy AVX-512 instructions.
On the Cannonlake chip you used, however, the i3-8121U, I don't measure any AVX-512 throttling at all - regardless of what instructions are running, up to and including dense 512-bit FMAs, the chip is running at the full 3.2 GHz (1 active core) or 3.1 GHz (2 active cores) turbo speed at all times.
This behavior was common in the past on client parts: e.g., most Skylake client parts run at full speed when using 256-bit AVX/AVX2 operations, but server parts and some higher-end desktop parts based on them had varying speed tiers. All of the initially released SKX (Skylake X) parts were based on the Skylake-SP server part and had the speed tiers (downclocking) for AVX-512, but maybe many or most future client parts won't if CNL is any indication.
That said, the biggest mystery in this article is why you'd ever want to remove all whitespace from text. Why is that useful?
sig nup now for che ap viag ra
{mystring:"Hello World!"}
{mystring:"Hello World !"}
They aren't the same JSON.
The code will have to be a lot more complicated in order to achieve this, and I'm not even sure I could use AVX512 to replace the ugly regex I'm currently relying on. But the idea is close enough, so maybe whitespace removal is meant to serve as a simple proof of concept.
So no, I don't think the transitions are a big problem for any throughput case, although if you are counting microseconds for some type of low latency thing then sure it can matter.
The bigger problem is usually the frequency reduction.
Gold or Platinum you're not likely to see much degradation from using AVX-512, even sparsely.
I am not aware of any strong variance in this behavior based on CPU class, and I would be surprised if it existed. I've always measured around 8 to 10 us for these transitions (there may be both a real time and "clock cycles" component to the transition time so it might vary a bit when measured in real time based on the CPU frequency).
It sounds to be like you are talking more about AVX-512 downclocking.
In any case, if your algorithm can be make heavy and effective use of AVX-512 it's going to be a win on almost any class of chip since you are looking at something like a 2x baseline speedup and more in many cases. If not, then yeah, proceed with caution.
Contrast:
Bronze: https://en.wikichip.org/wiki/intel/xeon_bronze/3106#Frequenc...
Silver: https://en.wikichip.org/wiki/intel/xeon_silver/4108#Frequenc...
Gold: https://en.wikichip.org/wiki/intel/xeon_gold/5118#Frequencie...
Platinum: https://en.wikichip.org/wiki/intel/xeon_platinum/8153#Freque...
FWIW, I recently tested a consumer Cannonlake chip and there is no downclocking even with heavy AVX-512: all types of tested code ran at the full turbo frequency. So it seems that not all chips will have these varying frequencies based on ISA extensions used.