Devil's advocate, the worst-case scenario is really bad, and would probably be common: A random button your pointer is unintentionally hovering on stays put while the rest of the page wildly formats around it, super-janky UX.
Alternatively, we could disable clicking/selecting anything that wasn't visible for at least 200 or 300 ms at that location. So if you try to click on a button and it's shifted down or something pops up in front, and you end up clicking on that instead, your cursor would just blink red.