Comment by theevilsharpie | Hacker News Reader

Comment by theevilsharpie | Hacker News Reader

This article isn't accurate, in the sense that it's actually measuring how long the operating system takes to reach the CPU's maximum clock speed, rather than how long it physically takes the CPU to reach that speed.

Modern processors can switch frequencies very fast -- generally within microseconds. An example from the machine I'm currently using:

    $ grep 'model name' /proc/cpuinfo | uniq; sudo cpupower frequency-info -y -m

    model name : Intel(R) Core(TM) i5-4310U CPU @ 2.00GHz
    analyzing CPU 0:
      maximum transition latency: 20.0 us

Even very deep CPU idle states can be exited in <1 ms.

With respect to the operating system, the amount of time it takes to reach maximum frequency from idle depends on:

- How frequently the OS checks to see if the frequency should be increased

- Whether the OS will step through increasing frequencies, or go straight to the max frequency

- If the OS is stepping through increasing frequencies, how many it needs to step through

- Whether the core is already active, or if it needs to be awakened from a sleep state first

It looks like the OP is using Windows. Windows has a number of hidden (and poorly documented) tunables that control the above settings, and would have a significant impact on how quickly a CPU can reach max frequency. Microsoft used to have an extensive document (specifically, a Word document for Windows Server 2008) describing how to tune these parameters using the `powercfg` CLI tool and provided a detailed reference for each parameter, which I unfortunately can't find anymore. It looks like Microsoft's modern server performance tuning documentation describes a subset of these parameters.[1]

Linux has similar tunables for its `ondemand` CPU scheduler.[2] It looks like the default sampling rate to determine whether to adjust the frequency is 1000 times greater than the transition latency, or about 20 milliseconds on my machine.

I'm not familiar with macOS, but it likely has something similar (although it may not be user-accessible).

[1] https://docs.microsoft.com/en-us/windows-server/administrati...

[2] https://www.kernel.org/doc/html/v5.15/admin-guide/pm/cpufreq...

Hey, author here. The Snapdragon 670, Snapdragon 821, and i5-6600K tests were done on Linux, and the rest were done on Windows. If Windows is delaying boost, it doesn't seem to be any different from Linux. And lack of intermediate states between lowest clock and max clock on all three of those (not considering the little cores) indicates the OS is not stepping through frequencies.

Since people don't usually write their own OS, I don't think it's correct to take the "maximum transition latency" reported by the CPU to mean anything, because users will never observe that transition speed. Also, processors that rely on the OS (no "Speed Shift") can transition very fast if their voltages are held high to start with, suggesting most of the latency comes from waiting for voltages to increase.

Please also read about Intel's "Speed Shift". While it's fairly new (only debuted in 2015), it means the CPU can clock up by itself without waiting for a transition command from the OS.

> Since people don't usually write their own OS, I don't think it's correct to take the "maximum transition latency" reported by the CPU to mean anything, because users will never observe that transition speed.

The Linux CPU frequency governor literally uses it as part of the algorithm for calculating its sampling rate (i.e., how frequently it checks whether to adjust the processor's frequency).

> Please also read about Intel's "Speed Shift". While it's fairly new (only debuted in 2015), it means the CPU can clock up by itself without waiting for a transition command from the OS.

Hardware-managed P-states don't need to wait for the OS to send a command to change the frequency, but the processor still performs periodic sampling to determine what the frequency should be (this happens ever millisecond on modern AMD and Intel hardware), the processor isn't necessarily going to choose the maximum frequency right away, and it's still subject to delays from the OS (e.g., Windows core parking).

In any case, a multi-millisecond delay in switching frequency isn't because the processor is waiting for the voltage to increase.

> The Linux CPU frequency governor literally uses it as part of the algorithm for calculating its sampling rate

Yes, the governor can play a role. It's visible to the user, which is the point. Also, the ondemand governor is actually irrelevant to the article as the S821 and S670 used the interactive and schedutil governors respectively, and the i5-6600K was using speed shift.

I think we're disagreeing because I really don't care about how fast a CPU could pull off frequency transitions if it's never observable to users. I'm looking at how it's observable to user programs, and how fast the transition happens in practice.

> Processor still performs periodic sampling...

Same as the above, that's not the point of the article. I'm not measuring "what could theoretically happen if you ignore half the steps involved in a frequency transition even though a user realistically cannot avoid them without serious downsides" (like artificially holding the idle voltage high and drawing more idle power, as in the Piledriver example)

> In any case, a multi-millisecond delay in switching frequency isn't because the processor is waiting for the voltage to increase.

Yes, there are other factors involved besides the voltage increase. I never said it was the only factor, and did mention speed shift taking OS transition commands out of the picture (implying that requiring OS commands does introduce a delay in CPUs without such a feature).

If you want to test how fast a CPU can clock up, without influence from OS/processor polling, please do so and publish your results along with your methodology. I think it'd be interesting to see.

> I think we're disagreeing because I really don't care about how fast a CPU could pull off frequency transitions if it's never observable to users.

I think we should care. What about interrupts that occur during that time? There are hardware devices that will just not work if it takes too long. Too long is usually 0.5 ms or so.

However, 20 microseconds is just fine.

That's a different and unrelated topic. If you're concerned about how fast device driver code can respond, well you can get a lot done in 0.5 ms even with the CPU running at 800 MHz or whatever the idle clock is.

Says someone who hasn't debugged a slow Windows graphics related ISR (interrupt), hogging the same CPU core where the your interrupt was supposed to be. (Also, whatever happened to that 50 µs ISR execution time limit? I guess it doesn't apply, if you're Microsoft. Then again, there was some GPU vendor code running as well in the call stack...)

0.5 ms is not really much on Windows.

I meant it's unrelated to how fast a CPU clocks up.

If something is taking longer than 0.5 ms, you shouldn't be doing it in the ISR. Queue up a DPC and do your longer running processing there, or send it to user space. And yeah it might not be your fault if another driver's ISR was hogging the CPU core. That's just a case of a badly written driver screwing up the world for everyone, because they're not supposed to be doing long running stuff in an ISR in the first place.

https://docs.microsoft.com/en-us/windows-hardware/drivers/de... says an ISR shouldn't run longer than 25 microseconds. 0.5 ms is an order of magnitude off. Not something going from 800 MHz to locked 4 GHz will fix.

My ISRs execute under 15 µs, some are as fast as 2 µs. I'm well aware of the DPC queueing.

> ISR shouldn't run longer than 25 microseconds

Weird, I think I read 50 microseconds somewhere else. Maybe I just remember it wrong?

> 0.5 ms is an order of magnitude off. Not something going from 800 MHz to locked 4 GHz will fix.

0.5 ms is actually not that far fetched with higher priority interrupts masking and the delay for Windows ISR dispatching. There are also SMM missing time black holes occasionally.

Windows isn't a real-time OS for sure! Although can't blame SMMs on Windows.

The interrupts are whatever, but the linux scheduler can do some really strange things with thrashing cpu frequencies, all while moving from core to core. So if I have a program running a core at 100%, the scheduler decides to move it to another core, meaning context switching, cache misses, interrupts, frequency scaling, etc. It adds up!

When my power supply isn't properly connected (i.e. partially but not fully), the CPU locks itself to 799.97MHz or so, until I reconnect it. Is that related to the idle clock you mention?

That’s happening because your BMC is asserting PROCHOT which sends the CPU to its lowest possible clock speed.

Modern CPUs do not pause during the frequency transition. They switch to a stable clock (even if slower) while waiting for the main PLL to relock at the new frequency.

I read the article in full, and the data & information was interesting, but I have to say a lot of the points you're making in the comments now was not clear from the article text alone.

Another important point is you're measuring the default behavior of the various control systems. One can change that which would allow the user to observe something else.

Yeah, didn't want to start an article with a five paragraph essay especially when wordpress pagination doesn't work, so I can't get an Anandtech style multi-page article up.

And yep. You can even run a CPU at full clock all the time, meaning you will never observe a clock transition time. Cloud providers seem to do that.

I actually liked the article and I think the first image is really informative.

I guess my point is the "clock frequency ramp time" is really due to the interplay of a bunch of different control systems, some in the OS and some not. And when those systems get mixed together, in a somewhat uncontrolled way (which is the case for most PCs), a huge amount of variability is the result and that's what the article did a good job quantifying but IMHO didn't make clear.

But at the time scales in your plots "how quickly CPUs change clock speeds" is basically an implementation choice.

Just my $0.02

Thanks :) I guess I can't reply to a 7th level comment, so hopefully this one shows up in the right place.

I agree, there are multiple factors at play. But I don't think it's basically an implementation choice. Certainly it looks like it in some cases (S821 on battery, HSW-E and SNB-E). But it doesn't seem to be the case elsewhere. For example, speed shift lowers clock transition time by taking the OS out of the control loop.

For some reason, this site hides the reply button after a certain reply-chain length, but you can just click on the person's name. This will show all their posts, including the one you want to reply to (you may have to look for it), with the reply buttons present.

I guess they must be trying to softly dis-incentivize really long chains, but not block them outright? It doesn't really make sense to me...

Faster is click on the timestamp of the post and you can reply directly in one click.

When a comment has the insight you expected from the article.

Thanks!

Reply on news.ycombinator.com