You need to account for the fact that a lot of scraping is delegated by the big players to smaller players who can take loss of reputation and can use questionable methods (residential IPs). Some of the scrapers from these AI companies have been written very poorly from performance perspective.

From what I understand the pressure has created even Google to be a lot more aggressive than what it was before. Not entirely sure but I believe google has two categories of scrapers, the regular one and a new one for AI