I created such a search engine myself, but instead of displaying the standard text results I displayed only the website favicons so as you scrolled down infinite favicons appear, click a favicon it takes you to the webpage.
I think nickporter hit the nail on the head about the first page of results being so important and as you mentioned the competition to be on Google's 1st page is good for Google financially. However, one complaint about my search engine was that since the results often repeated the same favicon users thought the results were subpar, never realizing the results were Google results all along. By limiting the results per page it is harder for people to see how often Google's search results repeat, which according to my feedback users associate with low quality results.
As you will immediately notice FB, Twitter, Whitehouse.gov, barackobama.com are repeated 3-4 times each, and this is only the 1st 30 results you could continue scrolling down infinitely and you are bound to see the repeat again and again.
My point was user feedback complained of repeated results on my search engine. Users generally thought my results were lower quality than Bing/Google despite being identical depending on the API I was using. Further, I believe a lot of this has to do with Google/Bing users being unable to see a large number of results on the 1st page.
But even linking to different subpages I think Google would prefer users not see multiple FB, Wikipedia, ect... results and that is just one rationale I offer for Google having an interest in limiting results/page (the question of the original submission).
That being said, it's interesting that I actually like it for the image page. I can't explain it - maybe it's just a "Well, for 15+ years I've used search this way" thing.
I use Google because the search results a much better.
However, in terms of UX, I personally feel that scrolling though an endless stream of results is much nicer than having them split up in groups of ten. I find it much nice to be uninterrupted when I'm trying to quickly scan through a ton of search results.
Let's say that you and I both search for a topic. The result we both want is result #33. We both find it, read it, and close the browser. The next day when we come back to the computer, we need to find it again. I think it's much easier to remember "That was at the top of the third page" than "I had to scroll maybe 5 or 6 times".
also, infinite scrolling is more for "discovering" without a specific result in mind whereas "searching" implies you have a result in mind and is either hit or miss.
Though it would be interesting to find out how many people do only check the first page before retrying their search.
According to this research http://searchengineland.com/organic-click-thru-rates-tumblin...
if google can't give you your desired results
within the first 10 results, they've failed.
I hate infinite scrolling. I don't know why people think it's a good idea. If you like it, reply to this comment and tell me why, because I really, totally don't see the value in it.
...I'd love someone to point out how out of date I am though, this is a year-old presumption! :)
he also states that an original reason was in order to optimize for speed rather than # of results, since people tend to modify keywords and research far more frequently than visiting the second page.
Disclaimer: I may be misattributing this, don't have the book with me.
Regardless, there is published data about how people click on the top 3 links the most, and click dropoff is precipitous after the second page. They simply don't need to do infinite scrolling...most people will requery the engine before checking the second or third page.
http://statspotting.com/2012/12/an-interesting-fact-about-go...
Plus, setting the cut-off at 1000 seems more of a "will not" than "can not".
Note: Google Instant gets turned off if you want more than 10 results per page.