Is Your Linux Version Hiding Interrupt CPU Usage from You? | Hacker News Reader

Is Your Linux Version Hiding Interrupt CPU Usage from You?

tanelpoder.com

133 pointstanelpoder16 comments

Irq accounting depends on architecture and code usage, e.g. like some kind of barrier (serializing) before the RDTSC.

on newer x86_64 the accounting code is around 100-200 cycles, so not a big deal, interrupt return and exit code is more than that (save context, load new context etc.).

The RDTSC and fence timings (Latency, Reciprocal throughput) [0]:

  AMD Zen 1
  SFENCE       4       ~20        
  LFENCE       1       0.25        
  MFENCE       7       ~70        
  RDTSC       37       36        
  AMD Zen 2
  SFENCE       1       1        
  LFENCE       1       11        
  MFENCE       7       ~78        
  RDTSC       37       37        
  AMD Zen 3
  SFENCE       1       1        
  LFENCE       1       10        
  MFENCE       6       ~60        
  RDTSC       44       36        

  Intel Haswell
  LFENCE       2      4        
  MFENCE       3     33        
  SFENCE       2      5        
  RDTSC       15     24        
  Intel Broadwell
  LFENCE       2      4        
  MFENCE       3     33        
  SFENCE       2      6        
  RDTSC       15     24        
  Intel Skylake
  LFENCE       2      4        
  MFENCE       4     33        
  SFENCE       2      6        
  RDTSC       20     25        
  Intel Skylake-X
  LFENCE       2      4        
  MFENCE       4     33        
  SFENCE       2      6        
  RDTSC       20     25        
  Intel Coffee Lake
  LFENCE       2      4        
  MFENCE       4     33        
  SFENCE       2      6        
  RDTSC       20     25

[0] https://www.agner.org/optimize/instruction_tables.pdf

I guess the fence instructions are there for avoiding the CPU out-of-order execution getting the timestamp too early (or late) compared to the interrupt service routine's start & end?

CONFIG_IRQ_TIME_ACCOUNTING according to the docs:

  CONFIG_IRQ_TIME_ACCOUNTING:

  Select this option to enable fine granularity task irq time
  accounting. This is done by reading a timestamp on each
  transitions between softirq and hardirq state, so there can be a
  small *performance impact*.
 
  If in doubt, say N here.

So I didn't enable this option in my kernels afraid of loosing performance on my laptop.

(Author here). It would be interesting to test the performance difference. I downloaded the source of one of my distro-provided kernels & recompiled it with different config setting (to compare perf with just one thing changed), but got into some trouble with my drivers and ran out of time.

I would say that on your laptop, you probably wouldn't even notice a difference (as the "rdtsc" timestamp reading operations are pretty efficient & done entirely within a CPU). But with anything that can interact with external hardware million(s) of times per second (heavy SSD I/O in servers and moving large amounts of data over network), may see a minor (but measurable) hit.

In my experience, when large amounts of CPU gets used by interrupt routines (either hardware or software interrupts), that more likely indicates some bug or configuration issue in a device driver or even some faulty hardware issue.

edit: so, running "dmesg" and checking hardware health metrics (mceloc, EDAC) is a reasonable thing to do when interrupt storms & related CPU usage suddenly show up.

Does this CPU time accounting have any noticeable performance impact?

I know in some embedded usecases, there are hundreds of thousands or millions of interrupts per second, and an interrupt routine might be only tens of clock cycles long. If that's the case, an extra bunch of instructions to time how long every single interrupt takes is a substantial overhead.

> Does this CPU time accounting have any noticeable performance impact?

Yes.

> Select this option to enable fine granularity task irq time accounting. This is done by reading a timestamp on each transitions between softirq and hardirq state, so there can be a small performance impact.

[0] https://cateee.net/lkddb/web-lkddb/IRQ_TIME_ACCOUNTING.html

Mods (dang): could we get the fragment specifier (#how-to-measure-interrupt-cpu-overhead-when-irq-time-accounting-is-disabled) removed from the URL?

Ahh, yes, that's actually a great point to be aware of, thanks @tanelpoder.

Looking at my machine: Funny to see 2h of interrupt processing over 12d uptime. Not that this is bad (0.80% on avg), but my first computer would have taken >6 weeks to process these interrupts - and not doing anything else. And that's only going by the clocks, a 80386 has a much worse IPC/pipeline than a 6th gen i7.

Thank you, this is something I somewhat recently realized I don't know how to measure and troubleshoot properly - I've never been able to give a confident answer to if context switching/interrupts are a bottleneck or not, just "the number looks highish and I don't have anything else to blame".

Can someone explain this in further detail. What is causing interrupt CPU usage on my machine. When I run powertop I see this:

  629 mW      2,6 ms/s     158,2        Interrupt      [0] HI_SOFTIRQ

Is this related or another story and what can I do about it?

Try disabling hardware devices until it goes away.

On a typical laptop, what level of interrupts is too high? And what can be done about it?

The tool 'powertop' is very useful here, it will show what is causing any form of wakeup on your system, and offers a 'Tunables' tab with suggested settings to improve some aspects.

Powertop is great. You can even run it in non-interactive mode. That said, it only optimizes for lower power usage and lower interrupts. You can further reduce interrupts by disabling devices. If there are devices you don't need, you can blacklist them in /etc/modprobe.d/ provided you understand the potential dependencies other devices may have on the drivers. This can free up a tiny bit of memory as well.

The truth is out there! The conspiracy is real!

Please don't do this here.

Reply on news.ycombinator.com