You are not supposed to wait until they are in the viewport. I use a 600px threshold so the visitor never sees anything on screen loading.

If your page has lots of search results and lots of images it really saves a huge amount and makes the page feel much lighter.