Once the page is rendered, the browser knows the (x,y,w,h) of every element in the DOM. It could store a reverse-map of that to quickly identify the DOM elements that contain any given (x,y), in the right nesting sequence. That could be used to optimize any pointer event. It could also have a special data structure just to identify DOM elements with :hover rules to specifically optimize hovers, which would help with both scrolling and just moving the mouse around the page.

There's probably a tradeoff between the time it takes to set up those data structures, the time they'd save, and the time it takes to handle the pointer and hover events without them. Maybe it's not worth it.