Let me put it a different way. There may exist two different programs that draw the same amount of power through the CPU but have different instructions-per-clock performance. The one with the higher IPC is getting all its extra instructions for free in energy terms.

Look at it another way: if a machine is already in its highest power state, with all cores running, but only executing one thread per core, running another thread on each core may cost almost nothing in terms of energy but could boost instructions retired by a lot.

The marginal cost of each instruction in compressed at the high end of utilization. The first instruction costs a lot, but after you're already running it the best thing you can do for efficiency is take utilization on that machine straight to 100%.

Datacenter grade stuff won't sleep as conservatively as a desktop grade CPU.

"access the RAM" is a big one that people overlook. Yes it might cost a few picojoules to load a line into cache, but the major cost of RAM is just that it exists and there is no low-power mode for refreshing it. RAM is incompressible and statically provisioned so if you can use memory more efficiently you can just buy less and this will save you a lot of power 24x7.