Hacker News Reader

If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions.

Is that pretty much normal?

View on HN

It's a courtesy, afaict.

Rather the opposite. Many robots.txt will disallow * but allow googlebot. This allows them to crawl those sites without getting flak for disregarding robots.txt completely.

Reply on news.ycombinator.com