Hacker News Reader

Comment by twoz | Hacker News Reader

Does Yahoo! not have a robots.txt file?

http://yahoo.com/robots.txt Sorry, the page you requested was not found.

If that page doesn't exist, then according to specification, they don't and you can crawl any page.

Update: Some Yahoo! subdomains have 'em.

Yes:

    http://search.yahoo.com/robots.txt
    http://groups.yahoo.com/robots.txt
    http://realestate.yahoo.com/robots.txt

No:

    http://maps.yahoo.com/robots.txt
    http://omg.yahoo.com/robots.txt