When you are comparing performance to kitty, are they both scrolling line by line (xterm's default), or jumping as with the xterm jumpScroll or fastScroll option?
I didn't get any difference with jumpScroll, but fastScroll makes it go further with xterm, though not as far and consistently as kitty.
So fastScroll option for xterm might improve the execution time.