What do you suppose we as website owners do to prevent our websites from being DoSed in the meantime? And how do you suppose we convince/beg the corporations running AI scraping bots to be better users of the web?
First of all web developers should use google and learn what a cache is. That way you don’t need compute at all.
It's funny really since Google and other search engines have been crawling sites for decades, but now that search engines have competition, sites are complaining.