ParentFull threadtompec·It does respect robots.txt when crawling. I'll add more details about this in the docs.View on HN