What do you do with these pages after you've crawled them? You need to build an index out of them, and serve that index out of some kind of low latency storage (DRAM, Flash). That makes increasing the index size very expensive. The index size has to be limited, and selecting the right pages to include in the index is thus a core quality feature for a search engine.
I still suspect that this whole thing is more about bias (and personalization, be it correct or incorrect) in the results.
[0] https://searchengineland.com/google-dumps-the-supplemental-r...
It's actually more complicated than just a single static index, which is also why it's unrealistic to expect a search engine to be deterministic at scale.
How would a search engine distinguish between the two kinds of queries, tens of thousands of times a second?
And how would one architect such a two-tiered system, particularly with an eye toward cascading failures?
I'd break up the indices into digestible chunks, perhaps chronologically by year/month crawled, and then run all queries simultaneously (in parallel) against all those index chunks and combine the results at the end. Infinitely scalable and can be tweaked to ensure specific response times.
And there'd definitely be no need to set some arbitrary date cut-off; just add a few more virtual machines. I'd bet that's what Google was doing, and then scaled back those machines to save money and boost profits.
Still, you can't keep partial results around forever, unless you want to make searches a lot more expensive, having to add a lot of capacity just to deal with the buffer bloat. Each query touches at least a thousand machines. Adding "a few more virtual machines" isn't going to cut it, especially if you have to handle tens of thousands of requests per second.
How much extra state (internal connections, memory for partial results, etc.) would such a new search type create?
How do you deal with a new kind of hot spots now?
What if millions of people suddenly activate such an option?
What if a botnet does it?