I've written crawler for all of the sites featured. Probably not the best/easiest way to do it but it works for me.