Viewport jumping is the scourge of web browsing; if some browser deployed some kind of advanced heuristic to try and get rid of it completely that alone might get me to switch.
Nearly all image formats have image dimensions in their header, so a hundred-byte prefetch per image should give the browser the image size IF it's not specified in the html or css.
I remembered hearing about Chrome doing it a while ago [1] but I still personally experience it fairly often.
Both Firefox and Chrome have implementations of scroll anchoring which is intended to fix this. The implementations are different and neither is perfect, but in a lot of cases they do help with jumping.