Hacker News Reader

Comment by peter422 | Hacker News Reader

For all the hate Google gets, it's crawlers are like 10x smarter than any other search engine. Wouldn't surprise me at all if there was some type of crawling issue that Google figured out and Bing didn't.

This is a case where I'm not ready to attribute to malice what is very likely incompetence.

View on HN

Operating 8.8.8.8 likely gives Google a near-complete list of every active domain on the internet, and new domains show up there the second any 8.8.8.8 user queries it - which is likely well before anyone links to it. I have no idea if Google use this data for hints to their crawler and indexer, but if I worked there I sure would.

It would seem very odd to not use this tool which effectively gives a huge competitive advantage over a lot of other engines, although for obvious reasons I doubt they'd shout it from the rooftops.

This actually makes perfect sense! I've always wondered why they operate this service. There's nothing in the privacy policy that suggests they won't use the aggregated anonymized information for other purposes.

I doubt they actually discover domains that way; for a search engine just knowing a domain exists isn't all that useful. Besides, loads of spam crawlers can find new domains plenty fast enough: create a brand new domain with a blank page and a brand new email address, link it once somewhere, and see how long it takes to get spam because some spam crawler bot found it.

The strange thing is that other search engines like Baidu and Yandex get this right...

Reply on news.ycombinator.com