Rules For Thee but Not for Me
because google (and the couple of other search engines) provide enough value that offset the crawler's resource consumption.
implying “robots.txt explicitly says i can’t scrape their site, well i want that data, so im directing my bot to take it anyway.”
Do you think this is such a horrible thing to scrape? I can't do it manually since there are few hundred locations. I could write some python script which uses playwrite to scrape things using my desktop browser in order to avoid CloudFlare. Or, which I am much more familiar with, I could write a python script that uses BeautifulSoup to extract all the relevant locations once for me. I would have been perfectly happy fetching 1 page/sec or even 1 page/2 seconds and would still be done within 20 minutes if only there was no anti-scraping protection.
Scraping is a perfectly legal activity, after all. Except thanks to overly-eager scraping bots and clueless/malicious people who run them there's very little chance for anyone trying to compete with Google or even do small scale scraping to make their life and life of local art enthusiasts easier. Google owns search. Google IS search and no competition is allowed, it seems.
Why is hammering the everloving fuck out of their website okay?
They made the data available on the website already, there's no reason to contact them when you can just load it from their website.