What does an idle CPU do? (2014) | Hacker News Reader

What does an idle CPU do? (2014)

manybutfinite.com

326 pointsivanvas101 comments

This is a good overview, but even by 2014 we weren't calling hlt - there's a few problems there. Imagine the situation where you fork() a process on one CPU, and you have another CPU that recently called hlt. Ideally you'd schedule it immediately instead of waiting for the next scheduler interrupt, but sending interrupts from one CPU to another (to break it out of hlt) is kind of expensive. The replacement is an instruction called mwait, which will (like hlt) wake the CPU up on interrupts, but will also wake up the CPU on a write to a specific memory address. If you want to wake up an idle CPU, the working CPU can just do that memory write and then the idle CPU will wake up back into the scheduler and figure out it has work to do.

(Edit: there are still actually some circumstances in which hlt is called rather than mwait, but this is very CPU dependent and is based on vendor recommendations. Also, before mwait showed up, switching to deep CPU power saving states tended to involve performing magic IO port accesses rather than calling hlt, and yes the world was worse back then)

Thanks for an awesome explanation. I always thought modern x86 uses IPI (Inter-Processor Interrupts) for that.

Is there a benefit of mwait vs IPI?

From the GP:

>[...] but sending interrupts from one CPU to another (to break it out of hlt) is kind of expensive. The replacement is an instruction called mwait, which will (like hlt) wake the CPU up on interrupts, but will also wake up the CPU on a write to a specific memory address.

> if the CPU is consistently busy for a normal user

It's probably Windows Store going bonkers with some insane backlog of periodic tasks, because you didn't open your laptop for a few months, that it absolutely at all costs must process for some reason, especially during a power outage when you're running on battery and need to use your laptop for once.

I agree. "Windows" and "emergency" don't go well together.

You can beat it into submission if you are okay with "purchasing" an enterprise license and set a bunch of GPOs and firewall rules. You can even go deeper and load unsigned drivers if you absolutely must.

I wonder how much of this will be lost if MS starts enforcing policies via attestation on its services.

On one hand I'd like to say I don't want to become a Windows admin to extract expected behaviour from a laptop, on the other that's exactly how I learned Linux administration.

Just uploading telemetry ;)

Oh, you’ve not moved your mouse for 5 seconds? Let me just defrag, build the windows search index and generally turn your laptop in to a portable heater.

In the days before MMU's and fancy DRAM controllers and newer DRAM chips the CPU had to do memory refresh during "idle" time.

Depends on what point in time you warp back to. There was a point where most home computers had static RAM.

Oh, look at Mr. Moneybags over here!

Yep but cost and when memory starting exceeding 64K.

Er, when? Apple II was sort of the first big successful home computer, and it had DRAM (The CPU and the display accessed the ram on alternate cycles, and Woz placed the display buffers so that most of the refresh was "free").

As far as I can tell, all the actually widespread successful computers after that have used DRAM too.

It's earlier than that, dating to the time of the Altair clones. Refresh hardware didn't come with those machines so adding it yourself complicated the memory board.

Yes, before the widespread successful bit, almost by definition, since SRAM was expensive.

https://en.wikipedia.org/wiki/Static_random-access_memory#In...: “SRAM was used for the main memory of most early personal computers such as the ZX80, TRS-80 Model 100 and Commodore VIC-20.”

It wouldn’t call it “widespread successful”, but he Macintosh Portable also had SRAM (https://en.wikipedia.org/wiki/Macintosh_Portable)

You are scaring me. Please, I need to be able to sleep later tonight. Warn before posting such a horror story!!!

Another horror story - I heard that a trick some software copy-protection (DRM) used back on the IBM PC is disabling the DRAM refresh interrupt. Every time you trigger the protection, you computer just crashes randomly.

I don't believe it. A quick search seems to indicate that the IBM PC 5150 triggered memory refresh via timer hooked up to an interrupt which would trigger the refresh hardware without cpu support.

There were computers before the IBM PC you know :).

Do you have one in mind? I'd love to read some.

The Apple II used DRAM (4KB was the minimum initially). There was no refresh logic. The CPU was slow (just a bit more than 1 MHz), and DRAM could perform two complete RAS cycles in that time. That allowed the design to perform one memory operation per CPU cycle and do one DRAM read for video display per cycle. The mapping of logical addresses to DRAM rows was designed such that during normal video display every row was touched frequently enough that it met refresh requirements.

Z80 CPUs had DRAM refresh logic built in. The first two clock cycles it would read the primary opcode byte, and the next two clock cycles it would do a refresh of the next row and then increment the row counter ("R"). That worked well because the minimum instruction took four clock cycles to complete. The timing was a bit surprising though -- it would seem to take four cycles to fetch the first byte of an instruction but only three clock cycles for each subsequent memory transaction. Unfortunately, the R counter was only 7b wide, which was fine for 16Kbit DRAMs, but became a problem for 64Kbit DRAMs which needed to refresh 256 rows, not just 128. Some Z80 based designs would decode the refresh cycles and either mux in their own 8b count value, or they ignored the Z80 and just stall the Z80 every so often to perform a refresh cycle.

I had an Ohio Scientific Challenger 1p (not totally sure on the numbers and letters) which I am pretty sure had 8K static RAM for main memory and a 1K video RAM for character addressable graphics. Microsoft BASIC in ROM and a tiny hex monitor for doing assembly. https://www.pcjs.org/machines/osi/c1p/

look up CP/M systems and or S-100 bus systems (they're generally the same machines)

A absolutely great visualization of this (or at least how the execution of an instruction by the CPU is affected by memory refreshing) is this video about the SNES. Video link has timecode, but it should be at 8:23.

https://youtu.be/Q8ph2OVqZeM?t=504

I find it really gives you some context. Memory is refreshed many, many times a frame, roughly once per scan line on a CRT.

Edit: Though it's unclear to me if it's the CPU actually doing the refreshing or if the CPU is just waiting until the refresh is complete.

Interesting. I understand why the CPU would have needed to do that in the absence of integrated DRAM controller but how does the absence or presence of an MMU relate to this?

I know you're referring to ancient architectures, but the question of when to run refresh commands is still surprisingly interesting and relevant for DRAM controllers today. Just to clarify, with DRAM controllers I mean the piece of hardware inside a CPU/SoC/Chipset that receives memory read/write requests and talks to the actual RAM via DDR2/3/4/5. The later standards get progressively more complex so I'm not familiar with them, but for DDR2/DDR3 you have a window of a few thousand memory clock cycles in which you need to issue a certain number of refresh commands IIRC. Since those refresh commands take a while (relatively speaking), it can be a good idea to run them when the number of outstanding commands is low - or some other strategy depending on context.

There are some more optimizations that DRAM controllers can perform. As part of reading a certain address, you first need to open a bank. This is not a free operation, it can take a while (relatively speaking). After the first read in a bank, you can read other addresses inside the same bank for free, without having to open the bank again. This means it makes sense for a DRAM controller to process reads out of order: Instead of processing a command sequence like A1 B1 A2 (A/B are banks here) in order, it saves time to process them out of order, i.e. A1 A2 B1.

Disclaimer: It has been a while since I last looked into DRAM controllers, so I hope I didn't miss anything. But the general ideas should be correct.

This is true and one of the reasons it's still not out of the question for bespoke memory controllers to be written for specific accelerators. There really isn't a one size fits all memory controller.

> do memory refresh during "idle" time.

Or in some real ancient case (ZX81), generate a video signal for the display (TV) - except in "fast" mode, where the screen would just show noise.

> the CPU had to do memory refresh during "idle" time

the CPU had to do memory refresh during busy times too!

The Z80 would do it automatically

2014 can be like stone age if you think a new kernel release gets released less than every 90 days. On the other hand development of some features just takes longer and longer than ever expected (not a surprise for any developer).

Well, stone age at least if you run a bit more fresh distro than some CentOS or oldstable Debian.

So are typical modern desktop Linuxes tickless already or not yet?

Answering to myself.

From Ubuntu 20.04 generic kernel 5.13:

   CONFIG_NO_HZ_COMMON=y
   # CONFIG_HZ_PERIODIC is not set
   CONFIG_NO_HZ_IDLE=y
   # CONFIG_NO_HZ_FULL is not set
   CONFIG_NO_HZ=y
   # CONFIG_HZ_100 is not set
   CONFIG_HZ_250=y
   # CONFIG_HZ_300 is not set
   # CONFIG_HZ_1000 is not set
   CONFIG_HZ=250

Without looking up all details now I guess CONFIG_NO_HZ_IDLE=y means tickless when idle?

That's of course not a very new kernel. Archlinux has 5.18, but don't have my machine accessible at this moment.

5.13 is pretty recent...in fact any 5.x is super up to date, still tons of 3.x and 4.x kernels running out there in the wild, wild real world.

3? Try 2.

Linux my-host-name-here 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux

Nearly all the (medical instrument) hardware where I work (a major hospital) is on 2.x, mostly running RHEL 5.

One would assume that medical device vendors could pay for extended support. However, even extended support for HEL 5 has ended now. And even before that in heavily regulated industry that uses such old stuff I would assume rolling out updates is a very slow progress.

Maybe not all that stuff connects to the internet, but I bet an ever increasing share does. So hospitals might have wose firewalls than others, so I would not surprised about such equipment being successfully attacked.

Of course old versions are unaffected because by newly introduced vulnerabilities. But old ones that have existed "forever" get detected. I guess it's not the kernal alone that is so old, but user space, too.

Arch, 5.18.7

  $ zgrep -E '^CONFIG\w*_HZ' /proc/config.gz
  CONFIG_NO_HZ_COMMON=y
  CONFIG_NO_HZ_IDLE=y
  CONFIG_NO_HZ=y
  CONFIG_HZ_300=y
  CONFIG_HZ=300

It's been that way for years, I remember seeing that since at least 2015.

LWN seems to confirm that feeling:

https://lwn.net/Articles/549580/

> In 3.10, the CONFIG_NO_HZ option has been replaced by a three-way choice:

> CONFIG_NO_HZ_IDLE (the default setting) …

https://en.wikipedia.org/wiki/Linux_kernel_version_history

3.10 30 June 2013

In DOS and Win3.x/9x, I believe the idle is instead an input polling loop. That's why VMs running those OSs will appear to be taking a full host core all the time, unless a special "idle TSR" utility is used.

I'm not sure if that polling instead of interrupts is also responsible for their perceived lower input latency relative to other OSs of the time.

Yes, I remember having to install a 3rd-party application to stop Windows 95 or 98 inside of a VM from using my entire host CPU.

Windows 95 didn’t use the HLT instruction because it didn’t work reliably on many PCs back in the day. It would sometimes freeze instead of just halt.

There was a post on the oldnewthing blog about this but I can’t immediately find it.

https://devblogs.microsoft.com/oldnewthing/20030828-00/?p=42...

Some of his posts are possibly more on the fictional side, but given that the Linux kernel also has (or had?) code to check for and also avoid a HLT instruction that doesn't work (search for "Checking 'HLT' instruction..."), I think this one is close to the truth (but of course, also begs the question of why MS didn't do what Linux did)

On the other hand, HLT was commonly found in the DOS software of the time, often for quick and approximate timing delays, and I don't remember any specific discussions about that causing problems, so I'm still not sure what the true problem is; my guess is that the combination of protected mode, which increases power consumption over real mode, and the sudden drop in current draw caused by a HLT, was enough to cause marginal power supply circuitry to fall out of regulation, similar to how stress-testing when searching for a stable overclock will sometimes crash the machine when it ends, not when it starts. On the more digital side, interactions with SMM might also be relevant: http://www.rcollins.org/ddj/Mar97/Mar97.html

> Linux kernel also has (or had?) code to check for and also avoid a HLT instruction that doesn't work

I wonder how this works. How do you safely check for an instruction that "locks ups unrecoverably" on some hardware? Will Linux hang on boot on such systems? That is still better than hanging randomly later, but I can see why Microsoft didn't want to go for that approach.

EDIT:

The check in Linux literally just called hlt a bunch of times and hoped it didn't crash. There was no recovery in case it failed (note how there's no code path to print anything else than "ok"). Searching for "checking hlt instruction" you can actually find people asking on forums why their boot stoped at that point. The check has been removed in recent kernels, so I had to go back in history a bit to find it.

https://github.com/torvalds/linux/blob/521cb40b0c44418a4fd36...

but I can see why Microsoft didn't want to go for that approach.

Yet Windows 95 has a message in its hardware detection dialog which tells you to restart the computer if it stops responding for a long time while it's detecting hardware --- and I believe it writes to the disk its progress, so it'll remember where it was and skip it the next time.

One wonders why neither MS nor Linux did that for HLT. Linux has a no-hlt boot parameter to disable its use.

I like the comment in the following function:

> If this fails, it means that any user program may lock the CPU hard. Too bad.

RISC-V: WFI - WFI (Wait For Interrupt) makes the processor suspend execution (Clock is stopped) until one of the following events take place: An IRQ interrupt. An FIQ interrupt. A Debug Entry request made to the processor.

ARM has the same. Apple's CPUs are interesting as they lose most register state on WFI, though there is a chicken bit to disable that behaviour - it's mentioned in one of the Asahi Linux blog posts [0]

[0] - https://asahilinux.org/2021/03/progress-report-january-febru...

I still don’t really see the need to wake up every 4ms when all processes are suspended or killed.

Just halt the CPU permanently until the next interrupt.

4ms interrupts are only needed when there are multiple active threads that need preempting.

I mean, a couple paragraphs down, the article says:

> The solution here is to have a dynamic tick so that when the CPU is idle, the timer interrupt is either deactivated or reprogrammed to happen at a point where the kernel knows there will be work to do (for example, a process might have a timer expiring in 5 seconds, so we must not sleep past that). This is also called tickless mode.

Modern kernels are tickless like you describe.

Back when computers had front panels full of lights they would often make pretty patterns with the lights when they had nothing else to do.

One of the fun things about building a recreation of the PDP-11/70 is to watch the idle display when RSX-11M+ isn't doing anything.

The RT-11 operation system for the PDP-11 was filled with amusing quotations all over the kernel source code.

The following is the source code of the CPU idle loop routine, with blinkenlights [0]. Note the funny quotations, and also note the use of self-modifying code - light direction reversal was implemented by overwriting the ROL/ROR instruction dynamically.

    ; "A source of innocent merriment!"
    ;   - W.S. Gilbert, "Mikado"
    ; "Did nothing in particular, and did it very well"
    ;   - W.S. Gilbert, "Iolanthe"
    ; "To be idle is the ultimate purpose of the busy"
    ;   - Samuel Johnson, "The Idler"
    ; "I got plenty of nothin', and nothin's plenty fo' me!"
    ;   - George and Ira Gershwin, "Porgy and Bess"
    ;-
    30$:
    .IF NE LIGH$T
        DEC     (PC)+           ;The RT-11 lights routine!
    LITECT:      .WORD  1
        BNE     ..NULJ          ;Not too often
        ADD     #<512.>,LITECT  ;Reset count, clear carry
    40$:        ROL     70$     ;Juggle the lights
        BNE     50$             ;Not clear yet
        COM     70$             ;Turn on lights, set Carry
    50$:        BCC     60$     ;Nothing fell off, keep moving
        ADD     #<100>,40$      ;Reverse direction
        BIC     #<200>,40$      ;ROL/ROR flip
    60$:        BIT     #<LIGHT$>,CONFG2 ;Does CPU have a light register?
        BEQ     ..NULJ          ;No
        MOV     (PC)+,@(PC)+    ;Put in lights (for 11/45)
    70$:         .WORD  0, SR
    .ENDC ;NE LIGH$T

If the code doesn't detect a light register, it runs the following NOP loop instead.

    ..NULJ::
    .IF EQ RTE$M
    ;;; WAIT                    ;Nothin' to do, so don't
        NOP                     ;Nothin' to do, so don't
    .IFF ;EQ RTE$M
        NOP                     ;Let the host do the waiting
    .ENDC ;EQ RTE$M
        NOP                     ;Second pad instruction
        .BR     SCNALL          ;Drop into ready job scan loop setup

[0] http://www.kpxx.ru/DEC/PDP-11/Software/OS/RT-11/05.07/05.07....

On my Windows laptop, Windows Defender is the devil waiting for idle hands.

My gosh this. I recently had Windows update on my sons laptop, a 2012 "hand me down" Thinkpad. We'd composed a song together in Ableton that used 4 soft-synths and after the update it could not play the song without pegging the CPU!

I eventually tracked this down to Windows Defender sitting at 40% CPU! Disabling it entirely brought the performance of the laptop back to what it was pre-update.

I find this behaviour extremely disappointing. This is a perfectly serviceable laptop that would have likely been replaced by someone who was less technically inclined had it happened to them and they did not have the wherewithal to track down the offending piece of software.

Your original quote reminded me of a Cormac McCarthy quote which seems to apply to the Windows Defender engineers right now (I kid):

> But when God made man the devil was at his elbow. A creature that can do anything. Make a machine. And a machine to make the machine. And evil that can run itself a thousand years, no need to tend it.

It's crazy because even slightly heavier Linux distros run smooth as butter on laptops that Windows chugs resources on. There's nothing fancier about Windows. There's literally no reason for it to be as poorly optimized as it is, and yet it is. It's crazy to me.

The reason is simple. It doesn’t impact Microsoft’s bottom line. Poor Linux performance will eventually annoy someone enough to fix it, and being open source they actually can fix it

It actually negatively impacts Microsoft's bottom line. A slow machine means a new PC purchased which usually means a new Windows licence, so there are negative incentives on Microsoft to improve Windows performance.

Or they switch to Apple and Microsoft is completely missing out on a customer.

Also the opposite end, with many CPU cores/threads such as the threadripper (2990wx, which is 32c/64 threads). It doesn't work under Windows 10 well at all, even with crazy hacks like coreprio https://bitsum.com/portfolio/coreprio/.

I've found Windows 11 out of the box works just as well as Linux. I never really need to dualboot Windows (I don't play as many games these days), but W11 looks like a lot of hard work has gone into making the scheduler and other tools work well. It's just a shame their UI is absolute garbage.

For me, W11 seems to have really messed things up with the scheduler. If I'm playing just about any game on my primary monitor, I can't leave a media player window or a browser window with youtube on the second monitor because Windows starves the (visible!) unfocused window if the focused one is using lots of resources. This makes the YouTube video I want to watch in the background stutter like crazy and often completely freeze until I refocus the window.

With games that support a gamepad and accept input while not focused, I've started leaving a browser window focused so that it gets correctly scheduled with essentially no noticable impact on the game.

There's a setting called game mode, turn that off.

That doesn't appear to help for me.

Even if you did play a lot more games, Steam's Proton has come a really long way, for most of my library playing Windows games under Linux is seamless now and almost seamless in VR too.

Whenever I want to game or do something resource intensive, I have to nuke windows defender 7 different ways in order to keep it from gobbling resources.

ELI5 : How nuke defender, and not have other risks? What method do you follow?

The last two straws that sent me to Linux on the desktop were Windows Defender's CPU usage and random, long-lasting periods of unexplained 100% disk usage.

The transition required a number of compromises and personal workflow re-engineering, but once that hump is overcome it's been smooth sailing. Only ever have to re-visit my dual-boot setup in Windows for the occasional android unlock tool.

Anyone know other resources like this? I am currently in the middle of learning about Operating Systems by reading my roommates old CS textbook and doing his projects. Would love to look at stuff like this because I just learned a chapter on interrupts and this supplemented it well.

Hennessey and Patterson contains a little bit of soft information/intuition like this (as well as the computer architecture)

Discussed at the time:

What Does an Idle CPU Do? - https://news.ycombinator.com/item?id=8529658 - Oct 2014 (55 comments)

Back in the day (8086) my idle loop would start at timer (2 sec) and if it expired without exiting idle, it would flush file system buffers. So if you powered off without shutting down, the file system(s) would be consistent.

Also every couple of seconds it would 'wash' an ECC memory page. So the machine wouldn't accumulate single-bit errors over weeks of uptime. The hardware didn't do that for you back then.

After putting that stuff in, our incidence of corrupt file systems went way down!

Can you tell us more about this system? ECC on an 8086 was rare.

CTOS running on a 'MegaFrame' which had 6 processors on cards the size of a pizza box. Slid them in a case the size of a dishwasher. Each had 1MB(!) of RAM, all mappable into each others' address spaces.

Also a desktop modular workstation, and NGen (Next Generations, hah). It went through 8086,186, 286 and 386 versions.

I'm surprised no one has mentioned Halt and Catch Fire - https://en.wikipedia.org/wiki/Halt_and_Catch_Fire_(computing...

Does anyone know if projects that aim to use CPU idle time are actually useful?

One example I found: https://einsteinathome.org/

Folding@home list of published results https://foldingathome.org/papers-results/?lng=en

I remember the seti one. Probably dedicated a decade of idle cpu usage to it.

(2014)

People make the argument that unused RAM is wasted RAM.

I wonder if the same logic applies to CPUs.

Unused clock cycles is wasted CPU.

Idle CPUs use much less power, so no its not wasted in that sense.

Let's say we have CPU with $500 cost which consumes 100W. 1 kWh is $0.1.

Case 1: we used this CPU for year with 1% average load. We paid $500 + 365 * 24h * 100W * 1% * $0.1 = $500 + $876 = $1 376.

Case 2: we used this CPU for year with 100% average load. We paid $500 + 365 * 24h * 100W * 100% * $0.1 = $500 + $87 600.

Case 3: like case 2, but we have cheap electricity with $0.01 for kWh. We will pay $500 + $8 760.

In any case CPU cost itself is tiny compared to energy price. So we can consider CPU cost is zero and all computations just require spending electricity.

So when some project asks to use your CPU at idle times, they're asking to use your electricity. And that's about it. You're donating electricity. It would be wrong to think that they're asking to use CPU which is not used anyway.

Now whether you want to donate electricity or not is another matter. But one should clearly understand what he does.

Edit: I forgot to divide by 1000, so those numbers are completely off, sorry.

> $500 + $87 600

100W * $0.10/kWh * 1 year = $87.66

You forgot to convert Wh to kWh. Google can evaluate the LHS in one step.

That aside, I agree. The price difference is even larger with 400W GPUs and data center bills which include cooling in the electricity cost. Initial cost matters very little vs efficiency.

Well, I guess that my conclusions were completely wrong for home usage. Spending $87 vs $8 per year is not a big deal and $500 CPU cost now dominates the total cost. So it makes sense to donate idle CPU time if you're fine with extra heat and some extra wear.

It was well worth making the point, though. I recently bought a 2nd hand Intel NUC as a potential home server. Came with Ubuntu and used about 10 watt on idle. Put OpenBSD on it and it now draws 20 watt - I assume because OBSD is much less aggressive about idling. That’s about £3 per month extra on electricity if run 24/7.

Ehh normally you’d want a home server to have a handful of SATA ports and enough PCIe lanes for a SAS card and GPU. If you’re spending 5W per HDD and hopefully in the future hundreds of watts for video transcoding, 20W for everything else is no big deal.

Anyway I’ve never used OpenBSD, but you can probably mess with the CPU governor settings to make it idle more aggressively.

IMO the “extra wear” is insignificant. Electromigration causes negligible wear compared to the thermal cycling of a typical gaming workload on a desktop GPU. Your conclusion was correct even if the math was off.

Afaik ram refreshing takes a good bit of power.

Makes more sense to disable unused modules imo.

And, indeed, going into deep processor power saving states these days will put the RAM into self refresh (which is one of the reasons that exit latency is greater for deep states)

It does. The author says "In fact, if the CPU is consistently busy for a normal user, it’s often a misconfiguration, bug, or malware." This is true for the personal computers he is discussing, but is not true for servers or mainframes. A lot of OS research back in the 70s was about how to maximize processor usage when handling many different jobs.

> This is true for the personal computers he is discussing, but is not true for servers or mainframes

Or for people who actually use their computers.

https://devblogs.microsoft.com/oldnewthing/20101203-00/?p=12...

> Unused clock cycles is wasted CPU.

It's definitely the case if your CPU is in the cloud and you're being charged by the minute.

>Whenever you write an if statement, Nyan Cat cries.

If I ever offended Nyan Cat, then I'm very sorry.

While I offend Nyan Cat,

Not sure if this loop ever exits.

Do until Nyan cat is offended

But why does Nyan cat get upset? Is this a bad nod towards branchless programming?

> branchless programming

On that note, why aren't we calling it "bamboo code"?

And maybe "spaghetti code" could be "kudzu code"?

It would seem appropriate, but important thing is that bamboo is branchless and can easily grow in parallel to other bamboos.

I think the author was implying if branches add complexity, which adds more points of failures. If your code doesn't branch, that's one less edge case to test :)

Reply on news.ycombinator.com