Show HN: Domain Crawler

github.com

2 pointscblock8112 comments

does this obey robots.txt and have some sort of delay so it doesn't take down a site?

It's a starting point for a crawler, and does not have limiters on robots.txt and sleeps yet. Users could easily add them in if want to crawl with it.

Reply on news.ycombinator.com