Here's the perf profile, just to clarify.
Cursor idle, viewport on the button examples. 3ms repaint every 8-11ms. Animation frame points to the code i mentioned.
The problem isn't doing work on a website, its if you have an idle task that is taking up a significant portion of the frame budget then its easy to have frame drops when you do significant work like clicking, scrolling, browsing.
Idle work showing up in flamecharts is usually the thing to be cautious of.