Creating a Robust, Reusable Link-Checker

I've done a similar project, it takes a website and output broken urls using Scrapy: https://gist.github.com/mdamien/7b71ef06f49de1189fb75f8fed91...