As someone who has done a tiny bit of scraping, I've been surprised how difficult it is to be a good citizen.
Are there packages you know of in the major language ecosystems that you can easily point at a both a robots.txt and an index.html (you need both to be compliant, most packages I've found only look at robots.txt) to get an answer on what you can and can't do at what rate?
The only one I'm aware of is a C# package which still requires you to do a lot of the heavy lifting[0].