HN
Hacker News
Top
New
Best
Ask
Show
Jobs
Nike.com's robots.txt file | Hacker News Reader
Nike.com's robots.txt file
nike.com
26 points
·
chirau
·
·
7 comments
Open article
Save
View on HN
antongribok
·
In Firefox I got access denied... In case anyone else can't get to it, here's a pastebin:
http://pastebin.com/raw/amTGwEEa
atombath
·
Not even disallowing crappy crawlers like majestic 12? They're going to scuff your Jordans!
JoshTriplett
·
Odd to prohibit crawling PDFs; modern search engines handle those just fine.
John23832
·
They probably just want to avoid having to serve them.
Ffaine
·
That's a nice work.
kombucha2
·
thats pretty great
ryanbertrand
·
just crawl it...expect SWFs and PDFs.
Reply on news.ycombinator.com