Hacker News Reader

Comment by charcircuit | Hacker News Reader

Google also keeps a copy of the "entire" web in RAM to search it faster.

Source? But it's definitely possible nowadays when RAM is cheap and modern Google makes the Internet feel incredibly smaller than it was 20 years ago.

At the time you would get tons of results from a myriad of small blogs, forums, niche websites. Nowadays it's Pinterest, blogspam and more SEO optimised algorithmic crap. If you're lucky you get a forum result that might actually be relevant.

Given the joke that if you can't Google it it doesn't exist, it follows that the Internet has considerably shrunk in the past two decades.

It looks like I was mistaken. They don't keep all of the documents in RAM, but I'm pretty sure they keep the search index (100 petabytes) in RAM. I'm struggling to find a source but I remember hearing it in a talk given by a Googler on YouTube. I also read somewhere that moving their whole index to RAM is how they brought their response times down from a couple of seconds to a couple hundred milliseconds.

Here, let Jeff Dean explain how it works (and why this is a complex problem that a lot of the naive 'build a better search engine than Google' miss)

https://pdfs.semanticscholar.org/fc32/72302461b74217662085a8...

The original article's description of how Google search signals work doesn't even scratch the surface of all the signals it's using.

A link to the talk for those curious

https://www.youtube.com/watch?v=IQuSuyUVkoE

Reply on news.ycombinator.com