A machine that is 99% idle isn't running at 1% utilization all the time. It's idling 99% of the time and running flat out 1% of the time. The user pushes at least one core to the limit with every action they take, even if only for microseconds.

This is why performance matters even with a machine that is mostly idle. A faster system will respond faster to events. It will have lower latency when communicating over a network. It will handle user inputs faster. Users will complain that a program takes a second to respond even if the load average is 0.01.

(I know CPUs enter lower-performance states when lightly loaded. They can go from idle to their highest performance state in much less time than it takes for the user to notice latency.)