Is it really 2x or is it 2x at max load ? Since, as you say, you're not peggig the CPU - would be interesting to compare power usage on a task basis and the duration. Could be that the 3D cache is really adding that much overhead even to idle CPU.

Anyway I've never regretted buying a faster CPU (GPU is a different story, burned some money there on short time window gains that were marginally relevant), but I did regret saving on it (going with M4 air vs M4 pro)