I'm not certain what causes it for you, but for me it was microsoft's antimalware scans. These are supposed to run only in 'idle' mode but for whatever reason were running while I was using my computer.
What's worse is the automatic OS restarts that seem to also think I'm in idle mode.