This is why I put the crucial space between "real" and "time". I cannot hold up to intensive buzzword scrutiny. :)
It's every 5 minutes because I don't want them to ban me. And in practice, this is an acceptable delay.
It's every 5 minutes because I don't want them to ban me. And in practice, this is an acceptable delay.
What you're doing is how it should be done for sources you don't have direct access to.
I'd use 10 minutes though, as http://www.uscourts.gov/robots.txt points to a crawl delay of 10. Whether it's checked is the risk you take.
http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl...