Similar thing for me. Pulseaudio sometimes takes a bunch of CPU with audio playing.

Even stranger, sometimes it keeps taking 5-10% CPU with no audio playing. (I think it is related to running out of memory for a moment and using swap, but no matter how much memory I free up, the computer never returns to normal and must be restarted).

I'm getting to the point of installing debug symbols and pointing perf at it, but dropping pulseaudio would be an even better solution.