Search era clearly proved it is possible to crawl respectfully - the AI crawlers have just decided not to. They need to be disincentivized from doing this
- is hard to enforce
- misses bite, i.e. it makes you more money to break it then any penalties
but in general yes, a site which indicates they don't want to be crawled by AI bots but still gets crawled should be handled similar to someone with house ban on a shop forcing them self into the shop
given how severely messed up some millennia cyber security laws are I wonder if crawlers bypassing Anubis could be interpreted as "circumventing digital access controls/protections" or similar, especially given that its done to make copies of copyrighted material ;=)
If you put something in public domain people are going to access it unless you put it behind a paywall but you don't want to do it because that would limit access or people wouldn't pay for it to begin with (ex. your blog nobody wants to pay for)
There's no law against scraping, and we've already past the CFAA argument
It's something people do for people. It's not "in the public domain" for companies to gobble up with machines.
> There's no law against scraping
There's no law against incurring as heavy and pernicious social and material costs to commercial scrapers as is physically possible within legal bounds, either. So what's the problem?