Hacker News Reader

Comment by adwf | Hacker News Reader

Looks cool, but I couldn't find any definitive answer about whether it obeys robots.txt or not? Just that it's upto the end-user to determine which pages get crawled.

I'm not too fussed about people crawling my sites (as you say, it's gonna happen anyway), but I do worry about certain dynamically built sections of websites that are off-limits to bots for good technical reasons.

View on HN

Although we do put the onus on the users to pay attention to robots.txt we realize the reality is that some amount of them won't necessarily respect it. That's one of the primary reasons behind the way we designed our crawler the way we did -- requiring people to actually specify the links it will visit (as opposed to spidering around sites following all links). Our hope at least is this requires people to put a little thought into the data they want (and where they want it from) before hitting a site.

Sounds good enough to me. Might have to keep an eye out for it, hopefully your users will all be good ;)

Maybe you could design a feature that gives them a little friendly reminder if they're about to cross someones robots.txt, lets them check that they're aren't about to do something seriously annoying. Sometimes a robots.txt is over zealous and should be crossed, most of the time not though.

Ah, yeah that's not a bad idea. Still keep the responsibility on the user, but do our best to keep them honest :)

I believe I remember you previously writing that you're also going to allow site owners to come to login to kimono and turn off spidering for their site?

Yes, absolutely... we'll be providing more control for webmasters before we exit beta.

Reply on news.ycombinator.com