To clarify a bit: the passive event listener idea is a good start because it means the scrolling thread doesn't have to block on the main JS thread.

But it seems like a big use of scroll handlers is to reposition some elements on the page, e.g. for parallax scrolling. Ideally that should synchronize exactly with the rendering thread, so elements don't jiggle slightly out of position as you scroll. The best way to do that is to run the scroll handler in the scrolling thread -- as long as you can avoid waiting for another thread.