Hacker News Reader

Comment by rms | Hacker News Reader

Google indexes this site very fast. I think it probably happens within 5 minutes.

Let's give it a try. It's 10:26PM EST.

This is a very obscure phrase that no one else ever uses.

There are at least tens of millions of web sites. This site ranks 3500 on Alexa and 12k on Compete. Google knows that and prioritizes its crawlers accordingly.

Maintaining a fresh index of the top 100k sites is where search engines get their best bang for the buck.

Doesn't Google use Pagerank for crawling policy/priority?

2 minutes later, it's there. This is a very obscure phrase that no one else ever uses. ... hackerne.ws/item?id=973578 - 1 minute ago

Damn.

Going even further off-topic, the sheer speed of Google's updates is one reason why I have felt that the pain-point of "real-time search" is overrated.

If you were able to sort the results by "last updated" then I might agree, but AFAIK you can't.

With Twitter Search you can easily see the most recent results first.

You can sort by date in a google search. Click on "Show Options" just below the search field, and about 15 lines down in the left bar, "Sorted By Date."

I think it's a fairly recent feature, but it's there.

There's a difference between searching for a unique phrase (the document is the only exemplar, therefore it ranks #1) and searching for the best page on a breaking topic.

That said, it's not clear to me that humanity actually needs real-time search. Maybe there's some advertising value in winning the race to be #1 for "hudson plane crash" but is there really a business here?

That the hackerne.ws domain is what gets indexed is so fucking ridiculous:

  1) Some cretin camped on a domain and didn't have the courtesy to do a 301
  2) PG is an asshole and ignores the Host header. HTTP/1.1 was a decade ago!
  3) Google is a jackass and prefers a low-pagerank domain.
     Maybe some asshat used Feedburner to proxy RSS via that domain?

That domain is no longer the one indexed. It knows.

Google offers a "push" mechanisms for sites. You'll have to google for the specifics, but if HN implements it then google knows when a page is updated and re-reads just that page.

I wonder if this site uses it though. I would guess, it probably does.

Reply on news.ycombinator.com