the problem usually looks more like this: I load a page and start reading. When I'm done reading the text/content in the viewport, I scroll down. On a lazily loaded page, now it needs to download the content in the new viewport position. On a non-lazily loaded page, the time I took to parse the initial content allowed the browser to complete loading the page. So effectively the lazily loaded page is taking the 10 seconds it takes to load the page, and rather than doing it while I'm occupied on other things, it holds on to most of that ten seconds, and spends that time when it interferes with what I'm doing, paying it out--2 second wait here, 2 second wait there.
Obviously this is an implementation detail and well done LL could keep loading everything until its done, but frequently it doesn't try until I scroll down. NonLL pages also aren't perfect, and if a page tries to load everything at once, each thing added slows everything else down. But for some reason I never experience that on non-LL pages (or maybe they're good LL pages and I don't notice.) It could be that browsers do some smart things (the obvious would be something like queueing requests for page content, only allowing 5 or so open requests, and loading images in the order they are referenced in the HTML so stuff at the bottom of the page is loaded last.)