There was an excellent talk from Jeff Dean a while back which went into a lot of detail.
Or this one: https://www.youtube.com/watch?v=modXC5IWTJI (terrible audio)
Both are very old, but extremely well aged.
IMO, the secret behind Google Search is not the smartness of the algorithms, but how much of it is baked in those O(100ms) which takes Google machines to answer your query. That's why the links above are the true reason Google Search performs well.
Natural Language Processing and Information Retrieval state of the art is far beyond what Google/Bing/Yahoo/Yandex/Baidu employ. But, it's far too expensive to serve it at QPS & latency required for decent UX.
Then again maybe i'm underestimating how often I don't know exactly what I'm looking for and just grab the first result.
Or is that not how it's done, and each search, for a given person, follows the same process?
I've always been curious about that.
While there's some very short caching of results, to my understanding, there's generally still going to be a lot of hitting the index because there's just a lot of new information coming in all the time. We can't somehow store a set of results for say "cars" and figure it's going to be the same info from one minute to the next.
And results don't really have a lot of personalization for individuals. When you see differences, it's usually due to language and location.