Even with no audio at all?
Pulseaudio burning CPU during silence is usually due to an application holding open a channel that is either really playing silence or is keeping the Pulseaudio machinery awake. You could blame the apps, but IMHO that is definitely a Pulseaudio problem