Interesting, does your crawler check every link every hour, or does it go through them a batch at a time?