Hacker News Reader

Comment by ihiram | Hacker News Reader

The demo is magical. Excellent work!

One question about this FAQ:

   > Does ParseHub respect robots.txt?
   > We're working on an admin panel to give webmasters full transparency and control. We'll have more info soon.

What does that mean? I read it as ParseHub does not respect robots.txt, which as a content owner is a bit disappointing. Would you elaborate on your thinking?

View on HN

Sorry, we'll make the wording more clear.

At the moment, ParseHub does not respect robots.txt. We do expect to add this + features for webmasters in the future, but have not had the developer cycles to do this yet.

Please add this sooner than "in the future". Obeying the robots.txt is a _MUST_ for any robots out there. This also from a content owner. Thank you.

Reply on news.ycombinator.com