The most plausible reason I've seen was (I believe) in 'don't make me think' - The ideal search engine would return one result - exactly what you're looking for. If google isn't consistently return the relevant result in the top 10, it's already failed as a search engine. Research has shown that(iirc) <10% of users go past the first page of results, and over half of clicks are on the top 5 entries.

he also states that an original reason was in order to optimize for speed rather than # of results, since people tend to modify keywords and research far more frequently than visiting the second page.

Disclaimer: I may be misattributing this, don't have the book with me.

Google runs thousands of A/B tests...there is a good chance you've just never seen one.

Regardless, there is published data about how people click on the top 3 links the most, and click dropoff is precipitous after the second page. They simply don't need to do infinite scrolling...most people will requery the engine before checking the second or third page.